2023-03-24 16:53:33,449 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72 2023-03-24 16:53:33,467 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-24 16:53:33,485 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-24 16:53:33,485 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e, deleteOnExit=true 2023-03-24 16:53:33,485 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-24 16:53:33,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/test.cache.data in system properties and HBase conf 2023-03-24 16:53:33,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/hadoop.tmp.dir in system properties and HBase conf 2023-03-24 16:53:33,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/hadoop.log.dir in system properties and HBase conf 2023-03-24 16:53:33,487 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-24 16:53:33,488 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-24 16:53:33,488 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-24 16:53:33,602 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-24 16:53:34,026 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-24 16:53:34,031 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-24 16:53:34,032 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-24 16:53:34,033 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-24 16:53:34,033 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 16:53:34,034 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-24 16:53:34,034 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-24 16:53:34,034 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 16:53:34,035 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 16:53:34,035 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-24 16:53:34,036 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/nfs.dump.dir in system properties and HBase conf 2023-03-24 16:53:34,036 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/java.io.tmpdir in system properties and HBase conf 2023-03-24 16:53:34,036 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 16:53:34,037 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-24 16:53:34,037 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-24 16:53:34,630 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 16:53:34,634 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 16:53:34,931 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-24 16:53:35,097 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-24 16:53:35,129 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 16:53:35,171 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 16:53:35,203 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/java.io.tmpdir/Jetty_localhost_34557_hdfs____.w0kfr4/webapp 2023-03-24 16:53:35,350 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34557 2023-03-24 16:53:35,392 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 16:53:35,392 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 16:53:35,854 WARN [Listener at localhost/40397] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 16:53:35,966 WARN [Listener at localhost/40397] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-24 16:53:35,991 WARN [Listener at localhost/40397] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 16:53:35,999 INFO [Listener at localhost/40397] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 16:53:36,004 INFO [Listener at localhost/40397] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/java.io.tmpdir/Jetty_localhost_38383_datanode____.cem08g/webapp 2023-03-24 16:53:36,107 INFO [Listener at localhost/40397] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38383 2023-03-24 16:53:36,503 WARN [Listener at localhost/43391] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 16:53:37,023 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbd2285c020fb7e86: Processing first storage report for DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1 from datanode 508011da-6d8b-49c9-a04a-1fbd55f43f65 2023-03-24 16:53:37,024 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbd2285c020fb7e86: from storage DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1 node DatanodeRegistration(127.0.0.1:39259, datanodeUuid=508011da-6d8b-49c9-a04a-1fbd55f43f65, infoPort=40055, infoSecurePort=0, ipcPort=43391, storageInfo=lv=-57;cid=testClusterID;nsid=1886292391;c=1679676814709), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-24 16:53:37,025 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbd2285c020fb7e86: Processing first storage report for DS-60c7f4fe-0989-4726-b27c-6593c47ec611 from datanode 508011da-6d8b-49c9-a04a-1fbd55f43f65 2023-03-24 16:53:37,025 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbd2285c020fb7e86: from storage DS-60c7f4fe-0989-4726-b27c-6593c47ec611 node DatanodeRegistration(127.0.0.1:39259, datanodeUuid=508011da-6d8b-49c9-a04a-1fbd55f43f65, infoPort=40055, infoSecurePort=0, ipcPort=43391, storageInfo=lv=-57;cid=testClusterID;nsid=1886292391;c=1679676814709), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-24 16:53:37,097 DEBUG [Listener at localhost/43391] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72 2023-03-24 16:53:37,159 INFO [Listener at localhost/43391] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e/zookeeper_0, clientPort=62195, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-24 16:53:37,171 INFO [Listener at localhost/43391] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62195 2023-03-24 16:53:37,178 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:37,180 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:37,847 INFO [Listener at localhost/43391] util.FSUtils(471): Created version file at hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641 with version=8 2023-03-24 16:53:37,848 INFO [Listener at localhost/43391] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/hbase-staging 2023-03-24 16:53:38,183 INFO [Listener at localhost/43391] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-24 16:53:38,689 INFO [Listener at localhost/43391] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-24 16:53:38,721 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:38,721 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:38,721 INFO [Listener at localhost/43391] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 16:53:38,722 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:38,722 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 16:53:38,868 INFO [Listener at localhost/43391] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 16:53:38,943 DEBUG [Listener at localhost/43391] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-24 16:53:39,037 INFO [Listener at localhost/43391] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39701 2023-03-24 16:53:39,048 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:39,050 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:39,071 INFO [Listener at localhost/43391] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39701 connecting to ZooKeeper ensemble=127.0.0.1:62195 2023-03-24 16:53:39,111 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:397010x0, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 16:53:39,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39701-0x101177228be0000 connected 2023-03-24 16:53:39,143 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 16:53:39,144 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 16:53:39,147 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 16:53:39,155 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39701 2023-03-24 16:53:39,156 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39701 2023-03-24 16:53:39,157 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39701 2023-03-24 16:53:39,157 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39701 2023-03-24 16:53:39,158 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39701 2023-03-24 16:53:39,164 INFO [Listener at localhost/43391] master.HMaster(444): hbase.rootdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641, hbase.cluster.distributed=false 2023-03-24 16:53:39,242 INFO [Listener at localhost/43391] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-24 16:53:39,243 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:39,243 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:39,243 INFO [Listener at localhost/43391] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 16:53:39,243 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 16:53:39,243 INFO [Listener at localhost/43391] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 16:53:39,249 INFO [Listener at localhost/43391] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 16:53:39,252 INFO [Listener at localhost/43391] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33315 2023-03-24 16:53:39,255 INFO [Listener at localhost/43391] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-24 16:53:39,263 DEBUG [Listener at localhost/43391] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-24 16:53:39,264 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:39,267 INFO [Listener at localhost/43391] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:39,268 INFO [Listener at localhost/43391] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33315 connecting to ZooKeeper ensemble=127.0.0.1:62195 2023-03-24 16:53:39,273 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:333150x0, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 16:53:39,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33315-0x101177228be0001 connected 2023-03-24 16:53:39,274 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 16:53:39,276 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 16:53:39,277 DEBUG [Listener at localhost/43391] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 16:53:39,280 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33315 2023-03-24 16:53:39,280 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33315 2023-03-24 16:53:39,281 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33315 2023-03-24 16:53:39,282 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33315 2023-03-24 16:53:39,283 DEBUG [Listener at localhost/43391] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33315 2023-03-24 16:53:39,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:39,294 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 16:53:39,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:39,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 16:53:39,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 16:53:39,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:39,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 16:53:39,316 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39701,1679676818001 from backup master directory 2023-03-24 16:53:39,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 16:53:39,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:39,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 16:53:39,321 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 16:53:39,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:39,325 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-24 16:53:39,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-24 16:53:39,415 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/hbase.id with ID: 3c1976f9-0e8f-49bd-b072-ba2f8cf8d6e8 2023-03-24 16:53:39,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 16:53:39,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:39,518 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x660fb749 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:39,552 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4af9001e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:39,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:53:39,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-24 16:53:39,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-24 16:53:39,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-24 16:53:39,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:53:39,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:53:39,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 16:53:39,642 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store-tmp 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 16:53:40,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:53:40,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 16:53:40,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/WALs/jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:40,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39701%2C1679676818001, suffix=, logDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/WALs/jenkins-hbase4.apache.org,39701,1679676818001, archiveDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/oldWALs, maxLogs=10 2023-03-24 16:53:40,159 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK] 2023-03-24 16:53:40,168 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:53:40,234 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/WALs/jenkins-hbase4.apache.org,39701,1679676818001/jenkins-hbase4.apache.org%2C39701%2C1679676818001.1679676820108 2023-03-24 16:53:40,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK]] 2023-03-24 16:53:40,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:53:40,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:40,240 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,242 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,320 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,328 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-24 16:53:40,359 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-24 16:53:40,374 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:40,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,383 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,404 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 16:53:40,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:53:40,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65715195, jitterRate=-0.02076728641986847}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 16:53:40,410 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 16:53:40,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-24 16:53:40,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-24 16:53:40,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-24 16:53:40,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-24 16:53:40,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-24 16:53:40,468 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-24 16:53:40,468 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-24 16:53:40,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-24 16:53:40,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-24 16:53:40,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-24 16:53:40,531 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-24 16:53:40,533 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-24 16:53:40,539 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-24 16:53:40,545 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-24 16:53:40,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:40,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-24 16:53:40,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-24 16:53:40,562 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-24 16:53:40,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 16:53:40,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 16:53:40,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:40,569 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39701,1679676818001, sessionid=0x101177228be0000, setting cluster-up flag (Was=false) 2023-03-24 16:53:40,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:40,595 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-24 16:53:40,596 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:40,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:40,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-24 16:53:40,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:40,613 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.hbase-snapshot/.tmp 2023-03-24 16:53:40,687 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(952): ClusterId : 3c1976f9-0e8f-49bd-b072-ba2f8cf8d6e8 2023-03-24 16:53:40,691 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-24 16:53:40,697 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-24 16:53:40,697 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-24 16:53:40,701 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-24 16:53:40,702 DEBUG [RS:0;jenkins-hbase4:33315] zookeeper.ReadOnlyZKClient(139): Connect 0x3c3b4380 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:40,710 DEBUG [RS:0;jenkins-hbase4:33315] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32fcaa77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:40,711 DEBUG [RS:0;jenkins-hbase4:33315] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36c328f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-24 16:53:40,725 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-24 16:53:40,734 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33315 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-24 16:53:40,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,737 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679676850737 2023-03-24 16:53:40,738 INFO [RS:0;jenkins-hbase4:33315] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-24 16:53:40,738 INFO [RS:0;jenkins-hbase4:33315] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-24 16:53:40,738 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1023): About to register with Master. 2023-03-24 16:53:40,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-24 16:53:40,742 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39701,1679676818001 with isa=jenkins-hbase4.apache.org/172.31.14.131:33315, startcode=1679676819242 2023-03-24 16:53:40,743 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 16:53:40,743 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-24 16:53:40,750 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 16:53:40,750 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-24 16:53:40,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-24 16:53:40,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-24 16:53:40,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-24 16:53:40,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-24 16:53:40,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,765 DEBUG [RS:0;jenkins-hbase4:33315] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-24 16:53:40,766 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-24 16:53:40,768 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-24 16:53:40,768 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-24 16:53:40,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-24 16:53:40,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-24 16:53:40,775 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679676820773,5,FailOnTimeoutGroup] 2023-03-24 16:53:40,775 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679676820775,5,FailOnTimeoutGroup] 2023-03-24 16:53:40,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-24 16:53:40,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,788 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 16:53:40,790 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 16:53:40,790 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641 2023-03-24 16:53:40,804 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:40,808 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 16:53:40,811 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/info 2023-03-24 16:53:40,812 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 16:53:40,814 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:40,814 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 16:53:40,817 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/rep_barrier 2023-03-24 16:53:40,818 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 16:53:40,819 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:40,819 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 16:53:40,822 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/table 2023-03-24 16:53:40,823 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 16:53:40,824 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:40,826 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740 2023-03-24 16:53:40,827 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740 2023-03-24 16:53:40,832 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:53:40,833 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38757, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-24 16:53:40,834 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 16:53:40,839 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:53:40,840 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63208736, jitterRate=-0.05811643600463867}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:53:40,840 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 16:53:40,841 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 16:53:40,841 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 16:53:40,841 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 16:53:40,841 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 16:53:40,841 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 16:53:40,843 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 16:53:40,843 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 16:53:40,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,849 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 16:53:40,849 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-24 16:53:40,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-24 16:53:40,865 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641 2023-03-24 16:53:40,866 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40397 2023-03-24 16:53:40,866 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-24 16:53:40,871 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 16:53:40,871 DEBUG [RS:0;jenkins-hbase4:33315] zookeeper.ZKUtil(162): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,872 WARN [RS:0;jenkins-hbase4:33315] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 16:53:40,872 INFO [RS:0;jenkins-hbase4:33315] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 16:53:40,873 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/WALs/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,875 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33315,1679676819242] 2023-03-24 16:53:40,879 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-24 16:53:40,882 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-24 16:53:40,883 DEBUG [RS:0;jenkins-hbase4:33315] zookeeper.ZKUtil(162): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,894 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-24 16:53:40,903 INFO [RS:0;jenkins-hbase4:33315] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-24 16:53:40,924 INFO [RS:0;jenkins-hbase4:33315] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-24 16:53:40,928 INFO [RS:0;jenkins-hbase4:33315] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-24 16:53:40,928 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,929 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-24 16:53:40,936 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,936 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,936 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,936 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,937 DEBUG [RS:0;jenkins-hbase4:33315] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 16:53:40,938 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,938 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,938 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,953 INFO [RS:0;jenkins-hbase4:33315] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-24 16:53:40,955 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33315,1679676819242-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:40,970 INFO [RS:0;jenkins-hbase4:33315] regionserver.Replication(203): jenkins-hbase4.apache.org,33315,1679676819242 started 2023-03-24 16:53:40,970 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33315,1679676819242, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33315, sessionid=0x101177228be0001 2023-03-24 16:53:40,970 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-24 16:53:40,971 DEBUG [RS:0;jenkins-hbase4:33315] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,971 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33315,1679676819242' 2023-03-24 16:53:40,971 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:40,971 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33315,1679676819242' 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-24 16:53:40,972 DEBUG [RS:0;jenkins-hbase4:33315] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-24 16:53:40,973 DEBUG [RS:0;jenkins-hbase4:33315] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-24 16:53:40,973 INFO [RS:0;jenkins-hbase4:33315] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-24 16:53:40,973 INFO [RS:0;jenkins-hbase4:33315] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-24 16:53:41,034 DEBUG [jenkins-hbase4:39701] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-24 16:53:41,036 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33315,1679676819242, state=OPENING 2023-03-24 16:53:41,044 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-24 16:53:41,046 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:41,047 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 16:53:41,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:53:41,083 INFO [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33315%2C1679676819242, suffix=, logDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/WALs/jenkins-hbase4.apache.org,33315,1679676819242, archiveDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/oldWALs, maxLogs=32 2023-03-24 16:53:41,102 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK] 2023-03-24 16:53:41,105 INFO [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/WALs/jenkins-hbase4.apache.org,33315,1679676819242/jenkins-hbase4.apache.org%2C33315%2C1679676819242.1679676821087 2023-03-24 16:53:41,106 DEBUG [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK]] 2023-03-24 16:53:41,235 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:41,238 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-24 16:53:41,241 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-24 16:53:41,252 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-24 16:53:41,253 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 16:53:41,257 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33315%2C1679676819242.meta, suffix=.meta, logDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/WALs/jenkins-hbase4.apache.org,33315,1679676819242, archiveDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/oldWALs, maxLogs=32 2023-03-24 16:53:41,280 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK] 2023-03-24 16:53:41,285 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/WALs/jenkins-hbase4.apache.org,33315,1679676819242/jenkins-hbase4.apache.org%2C33315%2C1679676819242.meta.1679676821262.meta 2023-03-24 16:53:41,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39259,DS-e43665c5-d6c2-445b-a181-5ae2bc24f5a1,DISK]] 2023-03-24 16:53:41,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:53:41,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-24 16:53:41,305 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-24 16:53:41,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-24 16:53:41,316 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-24 16:53:41,316 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:41,316 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-24 16:53:41,316 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-24 16:53:41,319 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 16:53:41,321 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/info 2023-03-24 16:53:41,322 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/info 2023-03-24 16:53:41,322 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 16:53:41,323 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:41,323 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 16:53:41,325 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/rep_barrier 2023-03-24 16:53:41,325 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/rep_barrier 2023-03-24 16:53:41,325 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 16:53:41,326 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:41,327 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 16:53:41,328 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/table 2023-03-24 16:53:41,328 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/table 2023-03-24 16:53:41,329 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 16:53:41,330 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:41,332 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740 2023-03-24 16:53:41,335 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740 2023-03-24 16:53:41,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:53:41,342 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 16:53:41,344 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62153233, jitterRate=-0.07384465634822845}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:53:41,344 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 16:53:41,354 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679676821227 2023-03-24 16:53:41,372 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-24 16:53:41,373 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-24 16:53:41,373 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33315,1679676819242, state=OPEN 2023-03-24 16:53:41,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-24 16:53:41,377 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 16:53:41,382 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-24 16:53:41,382 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33315,1679676819242 in 327 msec 2023-03-24 16:53:41,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-24 16:53:41,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 524 msec 2023-03-24 16:53:41,393 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 734 msec 2023-03-24 16:53:41,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679676821394, completionTime=-1 2023-03-24 16:53:41,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-24 16:53:41,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-24 16:53:41,462 DEBUG [hconnection-0x64fc7b2c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:41,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59032, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:41,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-24 16:53:41,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679676881484 2023-03-24 16:53:41,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679676941484 2023-03-24 16:53:41,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 89 msec 2023-03-24 16:53:41,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39701,1679676818001-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:41,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39701,1679676818001-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:41,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39701,1679676818001-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:41,511 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39701, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:41,512 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-24 16:53:41,518 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-24 16:53:41,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-24 16:53:41,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 16:53:41,538 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-24 16:53:41,540 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:53:41,543 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:53:41,568 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:41,570 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1 empty. 2023-03-24 16:53:41,571 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:41,571 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-24 16:53:41,996 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-24 16:53:41,998 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8b973ca11f7cca1ffdda5919062ee7c1, NAME => 'hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 8b973ca11f7cca1ffdda5919062ee7c1, disabling compactions & flushes 2023-03-24 16:53:42,012 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. after waiting 0 ms 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,012 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,012 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 8b973ca11f7cca1ffdda5919062ee7c1: 2023-03-24 16:53:42,018 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:53:42,034 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679676822021"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676822021"}]},"ts":"1679676822021"} 2023-03-24 16:53:42,061 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:53:42,063 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:53:42,067 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676822063"}]},"ts":"1679676822063"} 2023-03-24 16:53:42,072 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-24 16:53:42,082 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8b973ca11f7cca1ffdda5919062ee7c1, ASSIGN}] 2023-03-24 16:53:42,084 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8b973ca11f7cca1ffdda5919062ee7c1, ASSIGN 2023-03-24 16:53:42,086 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=8b973ca11f7cca1ffdda5919062ee7c1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:53:42,237 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8b973ca11f7cca1ffdda5919062ee7c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:42,237 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679676822237"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676822237"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676822237"}]},"ts":"1679676822237"} 2023-03-24 16:53:42,244 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 8b973ca11f7cca1ffdda5919062ee7c1, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:53:42,404 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,404 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8b973ca11f7cca1ffdda5919062ee7c1, NAME => 'hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:53:42,405 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,405 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:42,405 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,405 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,408 INFO [StoreOpener-8b973ca11f7cca1ffdda5919062ee7c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,411 DEBUG [StoreOpener-8b973ca11f7cca1ffdda5919062ee7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/info 2023-03-24 16:53:42,411 DEBUG [StoreOpener-8b973ca11f7cca1ffdda5919062ee7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/info 2023-03-24 16:53:42,411 INFO [StoreOpener-8b973ca11f7cca1ffdda5919062ee7c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8b973ca11f7cca1ffdda5919062ee7c1 columnFamilyName info 2023-03-24 16:53:42,412 INFO [StoreOpener-8b973ca11f7cca1ffdda5919062ee7c1-1] regionserver.HStore(310): Store=8b973ca11f7cca1ffdda5919062ee7c1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:42,414 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,420 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:53:42,424 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:53:42,425 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8b973ca11f7cca1ffdda5919062ee7c1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64869333, jitterRate=-0.033371612429618835}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 16:53:42,425 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8b973ca11f7cca1ffdda5919062ee7c1: 2023-03-24 16:53:42,427 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1., pid=6, masterSystemTime=1679676822397 2023-03-24 16:53:42,432 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,432 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:53:42,434 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8b973ca11f7cca1ffdda5919062ee7c1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:42,434 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679676822433"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676822433"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676822433"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676822433"}]},"ts":"1679676822433"} 2023-03-24 16:53:42,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-24 16:53:42,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 8b973ca11f7cca1ffdda5919062ee7c1, server=jenkins-hbase4.apache.org,33315,1679676819242 in 194 msec 2023-03-24 16:53:42,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-24 16:53:42,446 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=8b973ca11f7cca1ffdda5919062ee7c1, ASSIGN in 359 msec 2023-03-24 16:53:42,447 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:53:42,448 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676822447"}]},"ts":"1679676822447"} 2023-03-24 16:53:42,451 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-24 16:53:42,455 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:53:42,458 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 926 msec 2023-03-24 16:53:42,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-24 16:53:42,543 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-24 16:53:42,543 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:53:42,580 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-24 16:53:42,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 16:53:42,605 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-03-24 16:53:42,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-24 16:53:42,626 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 16:53:42,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-24 16:53:42,643 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-24 16:53:42,645 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-24 16:53:42,646 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.324sec 2023-03-24 16:53:42,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-24 16:53:42,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-24 16:53:42,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-24 16:53:42,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39701,1679676818001-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-24 16:53:42,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39701,1679676818001-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-24 16:53:42,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-24 16:53:42,694 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x69cada93 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:42,699 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53e64c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:42,713 DEBUG [hconnection-0x6468bb18-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:42,724 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59040, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:42,738 INFO [Listener at localhost/43391] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:53:42,759 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=170, AvailableMemoryMB=6527 2023-03-24 16:53:42,776 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:53:42,780 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:53:42,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:53:42,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:53:42,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:53:42,798 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:53:42,800 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:53:42,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-24 16:53:42,803 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:42,805 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 empty. 2023-03-24 16:53:42,806 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:42,806 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:53:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 16:53:42,826 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:53:42,829 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d3c5b00db9700342afae5fb089181b24, NAME => 'TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d3c5b00db9700342afae5fb089181b24, disabling compactions & flushes 2023-03-24 16:53:42,841 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. after waiting 0 ms 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:42,841 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:42,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:42,846 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:53:42,848 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676822848"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676822848"}]},"ts":"1679676822848"} 2023-03-24 16:53:42,851 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:53:42,853 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:53:42,853 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676822853"}]},"ts":"1679676822853"} 2023-03-24 16:53:42,856 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:53:42,861 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, ASSIGN}] 2023-03-24 16:53:42,864 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, ASSIGN 2023-03-24 16:53:42,865 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:53:42,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 16:53:43,017 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d3c5b00db9700342afae5fb089181b24, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:43,017 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676823017"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676823017"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676823017"}]},"ts":"1679676823017"} 2023-03-24 16:53:43,021 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:53:43,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 16:53:43,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:43,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d3c5b00db9700342afae5fb089181b24, NAME => 'TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:53:43,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:53:43,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,185 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,188 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A 2023-03-24 16:53:43,188 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A 2023-03-24 16:53:43,190 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:53:43,190 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3c5b00db9700342afae5fb089181b24 columnFamilyName A 2023-03-24 16:53:43,191 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(310): Store=d3c5b00db9700342afae5fb089181b24/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:43,192 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,194 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B 2023-03-24 16:53:43,194 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B 2023-03-24 16:53:43,194 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:53:43,195 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3c5b00db9700342afae5fb089181b24 columnFamilyName B 2023-03-24 16:53:43,196 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(310): Store=d3c5b00db9700342afae5fb089181b24/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:43,196 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,198 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C 2023-03-24 16:53:43,198 DEBUG [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C 2023-03-24 16:53:43,198 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:53:43,199 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3c5b00db9700342afae5fb089181b24 columnFamilyName C 2023-03-24 16:53:43,200 INFO [StoreOpener-d3c5b00db9700342afae5fb089181b24-1] regionserver.HStore(310): Store=d3c5b00db9700342afae5fb089181b24/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:53:43,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:43,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:53:43,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:53:43,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d3c5b00db9700342afae5fb089181b24; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65230245, jitterRate=-0.02799360454082489}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:53:43,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:43,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., pid=11, masterSystemTime=1679676823174 2023-03-24 16:53:43,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:43,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:43,223 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d3c5b00db9700342afae5fb089181b24, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:43,223 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676823223"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676823223"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676823223"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676823223"}]},"ts":"1679676823223"} 2023-03-24 16:53:43,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-24 16:53:43,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 in 205 msec 2023-03-24 16:53:43,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-24 16:53:43,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, ASSIGN in 369 msec 2023-03-24 16:53:43,238 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:53:43,238 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676823238"}]},"ts":"1679676823238"} 2023-03-24 16:53:43,241 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:53:43,244 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:53:43,247 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 449 msec 2023-03-24 16:53:43,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 16:53:43,426 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-24 16:53:43,431 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x6d61f291 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,437 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f0e2360, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,449 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,452 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,455 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:53:43,458 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56816, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:53:43,466 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x02df095a to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,471 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7263b174, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,471 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,476 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,476 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,483 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,483 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,487 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,488 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,492 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,493 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,497 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,497 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,520 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,521 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,525 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,525 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:53:43,528 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:53:43,566 DEBUG [hconnection-0x2a27259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,566 DEBUG [hconnection-0x6de4daf9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,568 DEBUG [hconnection-0x32ace444-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,568 DEBUG [hconnection-0x3cbe814d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,571 DEBUG [hconnection-0x1d94e186-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,572 DEBUG [hconnection-0x7ab648ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,573 DEBUG [hconnection-0x3e1787eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,573 DEBUG [hconnection-0x5ad720d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,576 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,576 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50820, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,576 DEBUG [hconnection-0x156ee019-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:53:43,580 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50842, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,582 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50834, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,582 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50830, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:43,590 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50850, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,590 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50860, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,592 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,600 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50878, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:53:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:43,616 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:43,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:43,619 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:43,619 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:43,621 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:43,621 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:43,623 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:43,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:43,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:43,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:43,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:43,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:43,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:43,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:43,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:43,645 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:43,648 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:43,668 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 16:53:43,690 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:43,690 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:43,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:43,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:43,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:43,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:43,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:43,797 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/5990d0f43e424e959a4a0b1d32a969e2 2023-03-24 16:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676823734, max timestamp=1679676823804Num uniques 859; 2023-03-24 16:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676823734, max timestamp=1679676823804Num uniques 859; 2023-03-24 16:53:43,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:43,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676823734, max timestamp=1679676823804Num uniques 859; 2023-03-24 16:53:43,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:43,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679676823861, max timestamp=1679676823873Num uniques 1018; 2023-03-24 16:53:43,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 16:53:43,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676823862, max timestamp=1679676823875Num uniques 1018; 2023-03-24 16:53:43,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 16:53:43,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676823862, max timestamp=1679676823873Num uniques 1018; 2023-03-24 16:53:43,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 16:53:43,967 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/fb66407c8afd4217bed176deac17a274 2023-03-24 16:53:44,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676823924, max timestamp=1679676823944Num uniques 1127; 2023-03-24 16:53:44,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-24 16:53:44,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676823924, max timestamp=1679676823944Num uniques 1009; 2023-03-24 16:53:44,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12682, suffix heap size=132904,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=277 2023-03-24 16:53:44,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676823924, max timestamp=1679676823944Num uniques 1009; 2023-03-24 16:53:44,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 16:53:44,038 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/78dc3ea21ffd4fb9b288b633f1e41d14 2023-03-24 16:53:44,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/5990d0f43e424e959a4a0b1d32a969e2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2 2023-03-24 16:53:44,080 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2, entries=150, sequenceid=9, filesize=11.6 K 2023-03-24 16:53:44,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676884070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676824059, max timestamp=1679676824072Num uniques 1136; 2023-03-24 16:53:44,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 16:53:44,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676824059, max timestamp=1679676824074Num uniques 1136; 2023-03-24 16:53:44,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=6870, suffix heap size=146112,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=150 2023-03-24 16:53:44,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676824059, max timestamp=1679676824074Num uniques 1009; 2023-03-24 16:53:44,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 16:53:44,124 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/fb66407c8afd4217bed176deac17a274 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274 2023-03-24 16:53:44,176 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274, entries=150, sequenceid=9, filesize=11.6 K 2023-03-24 16:53:44,181 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/78dc3ea21ffd4fb9b288b633f1e41d14 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14 2023-03-24 16:53:44,190 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14, entries=150, sequenceid=9, filesize=11.6 K 2023-03-24 16:53:44,193 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=57.17 KB/58539 for d3c5b00db9700342afae5fb089181b24 in 525ms, sequenceid=9, compaction requested=false 2023-03-24 16:53:44,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 16:53:44,199 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:44,199 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:44,199 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:44,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,218 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,219 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,219 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:44,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=63.88 KB heapSize=175.92 KB 2023-03-24 16:53:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:44,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:44,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:44,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:44,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:44,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:44,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e64a4e2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:44,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:44,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,226 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:44,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:44,226 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:44,227 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:44,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,261 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,261 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:44,277 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:44,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:44,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:44,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:44,284 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:44,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d206de5a7fbf47d9a474b1327ba1713a 2023-03-24 16:53:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:44,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:44,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,354 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:44,354 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:44,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:44,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676884339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676884341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676884350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676884395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676884401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,454 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:44,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:44,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:44,500 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:44,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:44,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:44,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:44,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,508 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676884505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:44,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676884510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:44,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:44,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:44,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:44,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:44,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:44,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:44,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:44,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:44,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:44,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:44,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190e05cd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:44,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:44,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:44,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:44,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:44,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:44,580 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676884513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676884514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:44,586 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:44,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:44,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:44,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:44,586 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:44,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676884557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,597 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,614 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,614 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,614 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:44,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:44,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:44,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:44,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:44,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676884714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676884715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:44,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:44,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:44,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:44,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:44,743 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:44,743 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:44,744 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:44,744 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:44,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,747 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,747 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,748 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:44,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:44,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:44,776 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:44,776 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:44,776 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:44,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:44,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:44,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231d8f86[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:44,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:44,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:44,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:44,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:44,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:44,794 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:44,799 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:44,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:44,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:44,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:44,799 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:44,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676884791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676884793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676884792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,806 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:44,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,808 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,828 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:44,828 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:44,828 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:44,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:44,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:44,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,834 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:44,834 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:44,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:44,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.76 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/4c3d5f766b464206a8863a2e1d8fabe3 2023-03-24 16:53:44,934 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:44,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:44,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.76 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/d6e1238ce1c94bfc9330a03f7278a2f0 2023-03-24 16:53:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:44,979 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:44,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:44,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:44,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:44,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,988 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:44,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d206de5a7fbf47d9a474b1327ba1713a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a 2023-03-24 16:53:44,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:44,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:44,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:44,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:44,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:44,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:44,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:44,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a, entries=200, sequenceid=87, filesize=13.9 K 2023-03-24 16:53:45,006 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,006 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,006 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,008 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/4c3d5f766b464206a8863a2e1d8fabe3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3 2023-03-24 16:53:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f25f39e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676885024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,029 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3, entries=150, sequenceid=87, filesize=11.6 K 2023-03-24 16:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,035 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,035 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/d6e1238ce1c94bfc9330a03f7278a2f0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0 2023-03-24 16:53:45,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676885025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,059 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0, entries=150, sequenceid=87, filesize=11.6 K 2023-03-24 16:53:45,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~63.88 KB/65409, heapSize ~175.88 KB/180096, currentSize=114.05 KB/116790 for d3c5b00db9700342afae5fb089181b24 in 846ms, sequenceid=87, compaction requested=false 2023-03-24 16:53:45,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:45,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 16:53:45,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676824313, max timestamp=1679676825107Num uniques 859; 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676824313, max timestamp=1679676825107Num uniques 859; 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676824313, max timestamp=1679676825107Num uniques 859; 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:45,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:45,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:45,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:45,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:45,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/8c693ac5651545a4a285458485a02b4f 2023-03-24 16:53:45,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:45,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:45,176 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:45,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:45,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:45,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:45,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:45,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,188 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249d1f2c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,202 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,202 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,203 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,203 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,216 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,216 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,222 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,231 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679676825181, max timestamp=1679676825209Num uniques 859; 2023-03-24 16:53:45,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7005, suffix heap size=117064,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=153 2023-03-24 16:53:45,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676825181, max timestamp=1679676825208Num uniques 859; 2023-03-24 16:53:45,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 16:53:45,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676825181, max timestamp=1679676825208Num uniques 859; 2023-03-24 16:53:45,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 16:53:45,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,237 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,237 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679676825249, max timestamp=1679676825273Num uniques 1012; 2023-03-24 16:53:45,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46348, new segment data size=16848, suffix heap size=133216,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=368 2023-03-24 16:53:45,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679676825249, max timestamp=1679676825273Num uniques 1009; 2023-03-24 16:53:45,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=16848, suffix heap size=132904,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=368 2023-03-24 16:53:45,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16847, totalHeapSize=38512, min timestamp=1679676825249, max timestamp=1679676825275Num uniques 1009; 2023-03-24 16:53:45,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=16847, suffix heap size=132904,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=368 2023-03-24 16:53:45,337 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:45,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:45,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:45,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:45,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:45,350 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:45,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:45,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:45,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:45,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:45,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:45,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:45,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:45,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,362 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,362 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,362 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,362 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676885353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,366 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676885360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676885365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dd4561f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,381 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,381 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,383 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,383 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,394 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,395 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,395 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,409 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,409 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676885468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676885481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676885483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:45,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:45,523 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:45,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:45,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:45,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,527 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:45,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676885531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676885549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa819c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,563 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,564 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,574 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,575 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,575 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,589 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,589 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/77dd81a5aad74756891e6c67e02d3e3a 2023-03-24 16:53:45,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676885673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,689 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:45,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676885689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:45,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676885697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:45,716 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:45,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:45,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:45,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,720 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:45,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,725 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,725 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,725 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,725 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b33b5e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,739 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,751 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,751 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,753 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,753 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,768 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,768 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,868 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:45,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:45,882 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:45,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:45,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:45,884 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,884 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,886 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:45,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:45,890 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,890 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:45,891 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:45,891 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:45,893 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e1b3e5b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:45,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:45,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:45,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:45,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:45,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:45,900 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:45,908 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:45,908 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:45,908 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:45,908 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:45,908 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:45,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,916 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:45,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:45,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:45,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:45,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:45,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:45,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:45,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:45,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:45,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:45,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:45,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676885977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:45,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676885996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676886001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:46,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:46,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:46,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:46,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:46,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:46,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:46,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:46,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:46,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:46,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:46,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:46,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:46,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:46,037 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:46,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:46,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:46,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5190f329[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:46,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:46,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,046 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:46,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:46,048 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:46,055 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:46,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:46,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:46,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:46,055 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:46,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:46,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:46,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:46,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:46,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:46,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:46,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:46,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:46,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:46,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,065 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:46,065 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:46,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:46,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:46,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/b624564ba48b4d3f80f0fc1ae710e87c 2023-03-24 16:53:46,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/8c693ac5651545a4a285458485a02b4f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f 2023-03-24 16:53:46,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f, entries=200, sequenceid=109, filesize=13.9 K 2023-03-24 16:53:46,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/77dd81a5aad74756891e6c67e02d3e3a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a 2023-03-24 16:53:46,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a, entries=150, sequenceid=109, filesize=11.6 K 2023-03-24 16:53:46,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/b624564ba48b4d3f80f0fc1ae710e87c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c 2023-03-24 16:53:46,165 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:46,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:46,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c, entries=150, sequenceid=109, filesize=11.6 K 2023-03-24 16:53:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:46,178 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:46,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:46,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:46,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:46,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for d3c5b00db9700342afae5fb089181b24 in 1069ms, sequenceid=109, compaction requested=true 2023-03-24 16:53:46,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:46,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:46,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:46,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:46,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:46,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:46,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:46,183 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:46,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:46,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:46,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:46,188 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-03-24 16:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:46,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:46,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:46,189 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:46,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:46,189 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:46,191 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:46,191 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,191 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=34.8 K 2023-03-24 16:53:46,193 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fb66407c8afd4217bed176deac17a274, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679676823609 2023-03-24 16:53:46,195 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4c3d5f766b464206a8863a2e1d8fabe3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679676824072 2023-03-24 16:53:46,196 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 77dd81a5aad74756891e6c67e02d3e3a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824394 2023-03-24 16:53:46,198 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:46,213 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:46,213 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:46,213 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,214 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=39.4 K 2023-03-24 16:53:46,214 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5990d0f43e424e959a4a0b1d32a969e2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679676823609 2023-03-24 16:53:46,215 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d206de5a7fbf47d9a474b1327ba1713a, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679676824072 2023-03-24 16:53:46,216 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 8c693ac5651545a4a285458485a02b4f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824391 2023-03-24 16:53:46,240 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#10 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:46,248 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#11 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:46,336 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/fb4feee842994f2da7d748e54d32aace as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fb4feee842994f2da7d748e54d32aace 2023-03-24 16:53:46,382 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into fb4feee842994f2da7d748e54d32aace(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:46,382 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:46,383 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=13, startTime=1679676826182; duration=0sec 2023-03-24 16:53:46,383 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:46,383 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:46,396 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:46,396 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:53:46,397 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:46,397 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=34.8 K 2023-03-24 16:53:46,398 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 78dc3ea21ffd4fb9b288b633f1e41d14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679676823609 2023-03-24 16:53:46,399 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d6e1238ce1c94bfc9330a03f7278a2f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679676824072 2023-03-24 16:53:46,399 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b624564ba48b4d3f80f0fc1ae710e87c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824394 2023-03-24 16:53:46,448 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#12 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:46,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:46,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:46,489 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/6ce5163776954140a168c2cd8dacfb0e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ce5163776954140a168c2cd8dacfb0e 2023-03-24 16:53:46,503 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 6ce5163776954140a168c2cd8dacfb0e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:46,503 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:46,503 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=13, startTime=1679676826183; duration=0sec 2023-03-24 16:53:46,503 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:46,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676886502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676886507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676886509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676886551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676886558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676886613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,696 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/81defd914d9048acb53a38062f1fce4f 2023-03-24 16:53:46,708 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/fc382a2524d446fc8387c3943d9a833f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fc382a2524d446fc8387c3943d9a833f 2023-03-24 16:53:46,724 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into fc382a2524d446fc8387c3943d9a833f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:46,724 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:46,724 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=13, startTime=1679676826183; duration=0sec 2023-03-24 16:53:46,724 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:46,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:46,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676886818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:46,961 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:53:47,058 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-24 16:53:47,059 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-24 16:53:47,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:47,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676887123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,141 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/453e31cc1db44f82b37fcd1f83f0d29e 2023-03-24 16:53:47,173 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/69043a59f4aa461bab971168d5836428 2023-03-24 16:53:47,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/81defd914d9048acb53a38062f1fce4f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f 2023-03-24 16:53:47,194 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f, entries=150, sequenceid=163, filesize=11.8 K 2023-03-24 16:53:47,196 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/453e31cc1db44f82b37fcd1f83f0d29e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e 2023-03-24 16:53:47,208 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e, entries=150, sequenceid=163, filesize=11.8 K 2023-03-24 16:53:47,211 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/69043a59f4aa461bab971168d5836428 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428 2023-03-24 16:53:47,223 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428, entries=150, sequenceid=163, filesize=11.8 K 2023-03-24 16:53:47,230 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=26.84 KB/27480 for d3c5b00db9700342afae5fb089181b24 in 1041ms, sequenceid=163, compaction requested=false 2023-03-24 16:53:47,230 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:47,230 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:47,230 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:47,234 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b07ee6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:47,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:47,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:47,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:47,241 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:47,247 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:47,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:47,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:47,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:47,247 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:47,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:47,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,255 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:47,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:47,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,262 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,262 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:47,262 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:47,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:47,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:47,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:47,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:47,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:47,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:47,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:47,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:47,369 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:47,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:47,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:47,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:47,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:47,375 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:47,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:47,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:47,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:47,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:47,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:47,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:47,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:47,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:47,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:47,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:47,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:47,381 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:53:47,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:47,382 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:47,382 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:47,382 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:47,382 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:47,382 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:47,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:47,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676827549, max timestamp=1679676827555Num uniques 859; 2023-03-24 16:53:47,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:47,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676827549, max timestamp=1679676827555Num uniques 859; 2023-03-24 16:53:47,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676827549, max timestamp=1679676827555Num uniques 859; 2023-03-24 16:53:47,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:47,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676827577, max timestamp=1679676827599Num uniques 1018; 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676827577, max timestamp=1679676827599Num uniques 1018; 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676827577, max timestamp=1679676827599Num uniques 1018; 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:53:47,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:53:47,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676827618, max timestamp=1679676827647Num uniques 1077; 2023-03-24 16:53:47,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:53:47,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676827618, max timestamp=1679676827647Num uniques 1077; 2023-03-24 16:53:47,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676827618, max timestamp=1679676827647Num uniques 1077; 2023-03-24 16:53:47,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:53:47,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:53:47,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:47,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10796, totalHeapSize=24784, min timestamp=1679676827672, max timestamp=1679676827707Num uniques 1086; 2023-03-24 16:53:47,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=10796, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-24 16:53:47,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10796, totalHeapSize=24784, min timestamp=1679676827672, max timestamp=1679676827707Num uniques 1086; 2023-03-24 16:53:47,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=10796, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-24 16:53:47,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10796, totalHeapSize=24784, min timestamp=1679676827672, max timestamp=1679676827707Num uniques 1086; 2023-03-24 16:53:47,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=10796, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-24 16:53:47,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676887708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676827745, max timestamp=1679676827756Num uniques 1095; 2023-03-24 16:53:47,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676827745, max timestamp=1679676827756Num uniques 1095; 2023-03-24 16:53:47,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:53:47,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:53:47,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676827745, max timestamp=1679676827756Num uniques 1095; 2023-03-24 16:53:47,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676827786, max timestamp=1679676827799Num uniques 1054; 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676827786, max timestamp=1679676827799Num uniques 1054; 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676827786, max timestamp=1679676827799Num uniques 1054; 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:53:47,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:53:47,826 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/1f7d727936b84b0aa094efe9d45d5b3c 2023-03-24 16:53:47,871 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/98c43d8b0d4f48e8ac51c74b03e160e4 2023-03-24 16:53:47,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679676827864, max timestamp=1679676827911Num uniques 1013; 2023-03-24 16:53:47,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679676827864, max timestamp=1679676827911Num uniques 1013; 2023-03-24 16:53:47,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-24 16:53:47,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-24 16:53:47,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679676827864, max timestamp=1679676827911Num uniques 1013; 2023-03-24 16:53:47,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679676827927, max timestamp=1679676827937Num uniques 1172; 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679676827927, max timestamp=1679676827937Num uniques 1172; 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679676827927, max timestamp=1679676827937Num uniques 1172; 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-03-24 16:53:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=17032, suffix heap size=149856,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=372 2023-03-24 16:53:47,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:47,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676887964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676887965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:47,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676887966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676888068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676888068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676888072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,179 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 16:53:48,180 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-24 16:53:48,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676888270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676888276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676888278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,307 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/74e8d167e5354366abb00a467256e518 2023-03-24 16:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/1f7d727936b84b0aa094efe9d45d5b3c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c 2023-03-24 16:53:48,324 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c, entries=150, sequenceid=173, filesize=11.8 K 2023-03-24 16:53:48,332 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/98c43d8b0d4f48e8ac51c74b03e160e4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4 2023-03-24 16:53:48,339 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4, entries=150, sequenceid=173, filesize=11.8 K 2023-03-24 16:53:48,343 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/74e8d167e5354366abb00a467256e518 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518 2023-03-24 16:53:48,351 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518, entries=150, sequenceid=173, filesize=11.8 K 2023-03-24 16:53:48,354 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for d3c5b00db9700342afae5fb089181b24 in 973ms, sequenceid=173, compaction requested=true 2023-03-24 16:53:48,354 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:48,354 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:48,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:48,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:48,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:48,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:48,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:48,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:48,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb2dc85[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:48,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:48,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:48,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:48,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:48,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:48,367 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:48,372 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:48,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:48,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:48,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:48,372 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:48,374 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,374 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:48,374 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:48,377 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:48,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:48,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:48,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:48,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:48,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:48,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:48,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:48,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:48,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:48,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:48,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:48,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:48,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:48,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:48,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:48,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:48,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:48,491 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:48,491 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:48,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:48,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:48,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:48,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:48,494 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:48,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:48,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:48,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:48,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:48,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:48,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:48,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:48,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:48,501 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:48,502 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=161.02 KB heapSize=444.70 KB 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:48,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:48,539 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/4555bf79d9d24b1c80de5203f8717268 2023-03-24 16:53:48,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:48,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:48,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676888569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676888569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676888577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676888581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676888583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676888676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676888678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676888880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:48,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:48,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676888883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,000 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/138590e8ca3f4b2094349e61bcf08942 2023-03-24 16:53:49,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676889083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50842 deadline: 1679676889085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676889089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676889183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676889189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,429 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/35aed1c4b97343eb8ac0cda9f140dba9 2023-03-24 16:53:49,438 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/4555bf79d9d24b1c80de5203f8717268 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268 2023-03-24 16:53:49,446 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268, entries=150, sequenceid=330, filesize=11.9 K 2023-03-24 16:53:49,447 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/138590e8ca3f4b2094349e61bcf08942 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942 2023-03-24 16:53:49,455 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942, entries=150, sequenceid=330, filesize=11.9 K 2023-03-24 16:53:49,457 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/35aed1c4b97343eb8ac0cda9f140dba9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9 2023-03-24 16:53:49,466 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9, entries=150, sequenceid=330, filesize=11.9 K 2023-03-24 16:53:49,468 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~444.66 KB/455328, currentSize=26.84 KB/27480 for d3c5b00db9700342afae5fb089181b24 in 966ms, sequenceid=330, compaction requested=true 2023-03-24 16:53:49,468 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:49,468 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:49,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,472 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:49,473 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,473 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bbcdeae[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:49,475 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:49,475 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:49,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:49,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:49,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:49,478 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:49,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:49,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:49,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:49,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:49,484 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:49,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:49,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:49,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:49,484 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:49,491 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,491 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,491 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:49,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:49,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,494 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,498 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:49,498 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:49,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:49,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:49,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:49,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,499 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:49,500 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:49,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:49,600 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:49,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:49,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:49,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:49,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:49,607 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:49,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:49,608 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:49,608 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:49,609 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,609 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:49,610 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,611 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:49,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:49,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:49,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:49,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:49,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:49,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:49,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:49,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:49,616 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:49,616 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:49,616 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:49,616 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:49,616 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:53:49,616 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:49,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:49,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:49,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:49,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:49,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:49,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:49,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676829691, max timestamp=1679676829719Num uniques 859; 2023-03-24 16:53:49,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676829691, max timestamp=1679676829719Num uniques 859; 2023-03-24 16:53:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676829691, max timestamp=1679676829719Num uniques 859; 2023-03-24 16:53:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:49,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:49,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676829747, max timestamp=1679676829758Num uniques 1118; 2023-03-24 16:53:49,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-24 16:53:49,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676829747, max timestamp=1679676829758Num uniques 1118; 2023-03-24 16:53:49,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-24 16:53:49,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676829747, max timestamp=1679676829758Num uniques 1118; 2023-03-24 16:53:49,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676829816, max timestamp=1679676829833Num uniques 1077; 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676829816, max timestamp=1679676829833Num uniques 1077; 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676829816, max timestamp=1679676829833Num uniques 1077; 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:53:49,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:53:49,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:49,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676889871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676829857, max timestamp=1679676829872Num uniques 1136; 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676829857, max timestamp=1679676829872Num uniques 1136; 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676829857, max timestamp=1679676829872Num uniques 1136; 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 16:53:49,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 16:53:49,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676829924, max timestamp=1679676829964Num uniques 1045; 2023-03-24 16:53:49,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:53:49,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676829924, max timestamp=1679676829964Num uniques 1045; 2023-03-24 16:53:49,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:53:49,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676829924, max timestamp=1679676829964Num uniques 1045; 2023-03-24 16:53:49,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:53:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676830000, max timestamp=1679676830018Num uniques 1054; 2023-03-24 16:53:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-24 16:53:50,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676830000, max timestamp=1679676830018Num uniques 1054; 2023-03-24 16:53:50,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-24 16:53:50,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676830000, max timestamp=1679676830018Num uniques 1054; 2023-03-24 16:53:50,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-24 16:53:50,042 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/675a104d462c47a4a87592ca5f569237 2023-03-24 16:53:50,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679676830046, max timestamp=1679676830076Num uniques 1063; 2023-03-24 16:53:50,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-24 16:53:50,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679676830046, max timestamp=1679676830076Num uniques 1063; 2023-03-24 16:53:50,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-24 16:53:50,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679676830046, max timestamp=1679676830076Num uniques 1063; 2023-03-24 16:53:50,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-24 16:53:50,095 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/59fcd7e257024bfb99be58415d05388c 2023-03-24 16:53:50,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679676830105, max timestamp=1679676830116Num uniques 1122; 2023-03-24 16:53:50,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-03-24 16:53:50,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679676830105, max timestamp=1679676830116Num uniques 1122; 2023-03-24 16:53:50,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-03-24 16:53:50,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679676830105, max timestamp=1679676830116Num uniques 1122; 2023-03-24 16:53:50,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-03-24 16:53:50,137 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/418356c29ea34475be4c11f54797d79a 2023-03-24 16:53:50,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676890140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1679676830123, max timestamp=1679676830144Num uniques 1181; 2023-03-24 16:53:50,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=19736, suffix heap size=150792,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=431 2023-03-24 16:53:50,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1679676830123, max timestamp=1679676830144Num uniques 1181; 2023-03-24 16:53:50,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=19736, suffix heap size=150792,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=431 2023-03-24 16:53:50,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1679676830123, max timestamp=1679676830144Num uniques 1181; 2023-03-24 16:53:50,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=19736, suffix heap size=150792,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=431 2023-03-24 16:53:50,148 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/675a104d462c47a4a87592ca5f569237 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237 2023-03-24 16:53:50,156 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237, entries=100, sequenceid=337, filesize=9.5 K 2023-03-24 16:53:50,157 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/59fcd7e257024bfb99be58415d05388c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c 2023-03-24 16:53:50,165 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c, entries=100, sequenceid=337, filesize=9.5 K 2023-03-24 16:53:50,167 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/418356c29ea34475be4c11f54797d79a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a 2023-03-24 16:53:50,178 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a, entries=100, sequenceid=337, filesize=9.5 K 2023-03-24 16:53:50,180 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=147.60 KB/151140 for d3c5b00db9700342afae5fb089181b24 in 564ms, sequenceid=337, compaction requested=true 2023-03-24 16:53:50,180 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:50,180 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:50,183 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=154.31 KB heapSize=419.25 KB 2023-03-24 16:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:50,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de37d4b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676890188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,200 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,200 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676890193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676890194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676890195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/45ecad3f69814ea0b622ecedd853c0a7 2023-03-24 16:53:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,218 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,219 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:50,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676890247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50860 deadline: 1679676890301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1679676890312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676890312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676890312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,319 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:50,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:50,324 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:50,324 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:50,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:50,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,327 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,327 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:50,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:50,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:50,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:50,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:50,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:50,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:50,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59206f5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,340 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,342 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,342 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,347 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,347 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,358 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,358 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50824 deadline: 1679676890450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:50,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:50,463 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:50,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:50,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:50,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:50,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:50,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:50,470 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,470 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:50,471 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:50,471 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@238cc60a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,475 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,476 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,478 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,485 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,485 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,489 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,489 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,489 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,495 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:50,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,496 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676890505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676890514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676890515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676890515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,596 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:50,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:50,601 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:50,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,605 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:50,605 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:50,605 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,605 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,607 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:50,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:50,610 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,610 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:50,610 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:50,610 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,612 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea8357b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,618 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,628 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,629 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,633 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,638 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,638 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,638 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,638 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,638 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,638 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/9db4b1212aac4083887fceffb1614315 2023-03-24 16:53:50,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:50,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:50,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:50,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:50,745 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:50,745 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:50,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:50,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,751 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:50,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:50,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:50,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676890753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:50,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:50,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:50,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,759 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86700c0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,762 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,762 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,764 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,775 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,775 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,781 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,781 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,795 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,795 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,796 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,796 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,796 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,797 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,798 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,798 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,798 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,798 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676890812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676890820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:50,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676890822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676890823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:50,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:50,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:50,903 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:50,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:50,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:50,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,908 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:50,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:50,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:50,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:50,916 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5267cc04[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:50,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:50,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:50,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:50,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:50,926 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:50,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:50,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:50,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:50,926 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:50,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:50,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:50,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,933 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:50,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:50,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:50,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:50,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:50,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:50,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:50,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:50,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:50,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:50,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:50,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,047 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@171eecd4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/0a4339eb0bd740858a98ff9629020d27 2023-03-24 16:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,069 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,069 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,083 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,083 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,083 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:51,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/45ecad3f69814ea0b622ecedd853c0a7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7 2023-03-24 16:53:51,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7, entries=350, sequenceid=510, filesize=21.4 K 2023-03-24 16:53:51,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/9db4b1212aac4083887fceffb1614315 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315 2023-03-24 16:53:51,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315, entries=150, sequenceid=510, filesize=11.9 K 2023-03-24 16:53:51,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/0a4339eb0bd740858a98ff9629020d27 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27 2023-03-24 16:53:51,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27, entries=150, sequenceid=510, filesize=11.9 K 2023-03-24 16:53:51,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~439.13 KB/449664, currentSize=33.54 KB/34350 for d3c5b00db9700342afae5fb089181b24 in 939ms, sequenceid=510, compaction requested=true 2023-03-24 16:53:51,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:51,122 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:51,122 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:53:51,122 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:53:51,122 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:51,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:51,126 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 79939 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:53:51,126 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:51,126 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,126 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fb4feee842994f2da7d748e54d32aace, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=78.1 K 2023-03-24 16:53:51,126 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70175 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:53:51,126 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:51,126 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,126 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fc382a2524d446fc8387c3943d9a833f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=68.5 K 2023-03-24 16:53:51,127 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting fb4feee842994f2da7d748e54d32aace, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824394 2023-03-24 16:53:51,127 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fc382a2524d446fc8387c3943d9a833f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824394 2023-03-24 16:53:51,130 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 81defd914d9048acb53a38062f1fce4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679676825347 2023-03-24 16:53:51,130 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 453e31cc1db44f82b37fcd1f83f0d29e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679676825347 2023-03-24 16:53:51,130 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1f7d727936b84b0aa094efe9d45d5b3c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676826492 2023-03-24 16:53:51,130 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 98c43d8b0d4f48e8ac51c74b03e160e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676826492 2023-03-24 16:53:51,131 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4555bf79d9d24b1c80de5203f8717268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1679676827956 2023-03-24 16:53:51,131 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 138590e8ca3f4b2094349e61bcf08942, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1679676827956 2023-03-24 16:53:51,132 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 675a104d462c47a4a87592ca5f569237, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679676828564 2023-03-24 16:53:51,132 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 59fcd7e257024bfb99be58415d05388c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679676828564 2023-03-24 16:53:51,132 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 45ecad3f69814ea0b622ecedd853c0a7, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679676830157 2023-03-24 16:53:51,133 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9db4b1212aac4083887fceffb1614315, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679676830173 2023-03-24 16:53:51,154 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:51,159 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#28 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:51,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,187 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,192 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,192 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:51,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:51,223 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/97e1f655590f4611bb74a49f390a1898 2023-03-24 16:53:51,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:51,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,273 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/e1da7510d2ea497a8271c66c6608d35a 2023-03-24 16:53:51,297 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/422fe6d01f6246c999cea1ed6cdc7553 2023-03-24 16:53:51,305 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/97e1f655590f4611bb74a49f390a1898 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898 2023-03-24 16:53:51,312 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898, entries=150, sequenceid=518, filesize=11.9 K 2023-03-24 16:53:51,315 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/e1da7510d2ea497a8271c66c6608d35a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a 2023-03-24 16:53:51,335 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a, entries=150, sequenceid=518, filesize=11.9 K 2023-03-24 16:53:51,339 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/422fe6d01f6246c999cea1ed6cdc7553 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553 2023-03-24 16:53:51,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676831330, max timestamp=1679676831338Num uniques 859; 2023-03-24 16:53:51,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:51,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676831330, max timestamp=1679676831338Num uniques 859; 2023-03-24 16:53:51,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676831330, max timestamp=1679676831338Num uniques 859; 2023-03-24 16:53:51,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:51,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:53:51,350 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553, entries=150, sequenceid=518, filesize=11.9 K 2023-03-24 16:53:51,351 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=67.09 KB/68700 for d3c5b00db9700342afae5fb089181b24 in 159ms, sequenceid=518, compaction requested=true 2023-03-24 16:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:51,351 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:51,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=80.51 KB heapSize=216.23 KB 2023-03-24 16:53:51,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:51,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:51,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:51,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:51,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189210a1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,361 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,361 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/0b73ec470292401ca1488a851a3febd4 2023-03-24 16:53:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,370 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,370 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,381 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,382 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,382 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,382 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676891392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676891396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676891396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676891397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676891397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,482 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,487 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,488 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,488 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,489 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,498 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c37273[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,505 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676891501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676891506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676891506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50830 deadline: 1679676891508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676891509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,515 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,516 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,590 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/659ecd47a2af4f909ce886345dfa05ba as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/659ecd47a2af4f909ce886345dfa05ba 2023-03-24 16:53:51,597 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/86f36741a83844dbaefb8854e04eb0fd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86f36741a83844dbaefb8854e04eb0fd 2023-03-24 16:53:51,610 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into 86f36741a83844dbaefb8854e04eb0fd(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:51,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:51,610 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=10, startTime=1679676831122; duration=0sec 2023-03-24 16:53:51,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:51,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 16:53:51,618 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82361 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 16:53:51,618 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:53:51,618 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,619 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ce5163776954140a168c2cd8dacfb0e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=80.4 K 2023-03-24 16:53:51,619 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into 659ecd47a2af4f909ce886345dfa05ba(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:51,619 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:51,619 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=10, startTime=1679676831122; duration=0sec 2023-03-24 16:53:51,619 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6ce5163776954140a168c2cd8dacfb0e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679676824394 2023-03-24 16:53:51,619 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:51,621 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 69043a59f4aa461bab971168d5836428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679676825347 2023-03-24 16:53:51,621 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 74e8d167e5354366abb00a467256e518, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676826492 2023-03-24 16:53:51,622 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 35aed1c4b97343eb8ac0cda9f140dba9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1679676827956 2023-03-24 16:53:51,623 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 418356c29ea34475be4c11f54797d79a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679676828564 2023-03-24 16:53:51,624 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0a4339eb0bd740858a98ff9629020d27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679676830173 2023-03-24 16:53:51,624 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 422fe6d01f6246c999cea1ed6cdc7553, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679676830186 2023-03-24 16:53:51,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,637 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,639 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,640 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,641 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,641 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a89c83[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,660 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,660 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,660 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,663 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,663 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#34 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,669 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,669 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,709 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/b4faddb34a214b92b32a50063a5cf311 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b4faddb34a214b92b32a50063a5cf311 2023-03-24 16:53:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676891709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676891713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676891713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676891714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:51,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676891714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,722 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into b4faddb34a214b92b32a50063a5cf311(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:51,722 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:51,722 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=9, startTime=1679676831123; duration=0sec 2023-03-24 16:53:51,722 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:51,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/891092a6bcb146aa8aa925a01c27a8ec 2023-03-24 16:53:51,787 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,794 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b0a08e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,818 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,818 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,820 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,822 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,831 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,831 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,850 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:51,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:51,955 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:51,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:51,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:51,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:51,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:51,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:51,959 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,959 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:51,959 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:51,959 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:51,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2faba1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:51,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:51,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:51,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:51,970 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:51,973 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:51,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:51,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:51,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:51,973 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:51,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:51,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:51,983 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:51,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:51,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:51,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:51,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:51,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:51,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:51,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:51,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:51,984 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:51,984 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:52,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676892013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676892017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676892018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676892018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676892019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,084 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:52,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:52,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:52,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:52,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:52,089 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:52,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:52,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:52,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,094 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:52,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:52,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:52,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:52,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:52,096 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,096 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:52,098 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4c04bd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:52,102 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:52,102 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:52,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:52,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:52,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:52,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,105 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:52,109 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:52,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:52,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:52,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:52,109 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:52,111 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,112 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,112 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,112 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:52,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:52,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:52,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:52,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:52,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:52,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:52,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:52,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:52,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:52,223 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:52,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:52,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:52,224 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,224 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:52,225 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:52,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/079149308ead4dec8e72c46949d284e9 2023-03-24 16:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:52,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,234 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,234 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:52,235 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/0b73ec470292401ca1488a851a3febd4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4 2023-03-24 16:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f00df53[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:52,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:52,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,240 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:52,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:52,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4, entries=350, sequenceid=545, filesize=21.4 K 2023-03-24 16:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/891092a6bcb146aa8aa925a01c27a8ec as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec 2023-03-24 16:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 16:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:52,269 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:52,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:52,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:52,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:52,269 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:52,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/079149308ead4dec8e72c46949d284e9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9 2023-03-24 16:53:52,273 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,273 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,273 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:52,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 16:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:52,282 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~216.19 KB/221376, currentSize=107.34 KB/109920 for d3c5b00db9700342afae5fb089181b24 in 928ms, sequenceid=545, compaction requested=true 2023-03-24 16:53:52,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:52,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:52,282 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:52,282 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:52,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:52,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,284 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:52,284 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:52,284 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,284 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86f36741a83844dbaefb8854e04eb0fd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=45.5 K 2023-03-24 16:53:52,285 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:52,285 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:52,285 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,285 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/659ecd47a2af4f909ce886345dfa05ba, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.0 K 2023-03-24 16:53:52,286 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 659ecd47a2af4f909ce886345dfa05ba, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679676830173 2023-03-24 16:53:52,286 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 86f36741a83844dbaefb8854e04eb0fd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679676830173 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:52,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:52,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:52,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:52,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:52,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:52,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:52,290 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e1da7510d2ea497a8271c66c6608d35a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679676830186 2023-03-24 16:53:52,290 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 97e1f655590f4611bb74a49f390a1898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679676830186 2023-03-24 16:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:52,290 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 891092a6bcb146aa8aa925a01c27a8ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679676831349 2023-03-24 16:53:52,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:52,292 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0b73ec470292401ca1488a851a3febd4, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679676831332 2023-03-24 16:53:52,307 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:52,308 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:52,344 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/5d4d1b8d772b4ecd8b92b6b4934c72c4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5d4d1b8d772b4ecd8b92b6b4934c72c4 2023-03-24 16:53:52,344 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/7a451b5af9d1494e91e76e4168821595 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7a451b5af9d1494e91e76e4168821595 2023-03-24 16:53:52,356 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into 5d4d1b8d772b4ecd8b92b6b4934c72c4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:52,356 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:52,356 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=13, startTime=1679676832282; duration=0sec 2023-03-24 16:53:52,356 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:52,357 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 16:53:52,357 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into 7a451b5af9d1494e91e76e4168821595(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:52,357 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:52,357 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=13, startTime=1679676832282; duration=0sec 2023-03-24 16:53:52,357 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:52,358 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 16:53:52,358 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 16:53:52,358 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. because compaction request was cancelled 2023-03-24 16:53:52,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:52,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:52,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:52,396 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:52,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:52,397 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:52,397 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,397 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:52,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:52,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:52,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:52,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:52,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:52,412 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:52,412 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:52,412 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:52,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:52,488 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/46d345d2d9d142c18671f4732191cac8 2023-03-24 16:53:52,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:52,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676892532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676892534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676892537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676892539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676892540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676892643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676892643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676892643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676892646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676892648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676892847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676892847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676892849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676892853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676892851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:52,969 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/de9e25c468704651baecd5ece7bbdba7 2023-03-24 16:53:52,992 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/d35317ba8cff431aa17149c747973891 2023-03-24 16:53:53,000 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/46d345d2d9d142c18671f4732191cac8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8 2023-03-24 16:53:53,010 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8, entries=150, sequenceid=569, filesize=11.9 K 2023-03-24 16:53:53,011 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/de9e25c468704651baecd5ece7bbdba7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7 2023-03-24 16:53:53,018 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7, entries=150, sequenceid=569, filesize=11.9 K 2023-03-24 16:53:53,019 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/d35317ba8cff431aa17149c747973891 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891 2023-03-24 16:53:53,031 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891, entries=150, sequenceid=569, filesize=11.9 K 2023-03-24 16:53:53,032 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d3c5b00db9700342afae5fb089181b24 in 620ms, sequenceid=569, compaction requested=true 2023-03-24 16:53:53,032 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:53,032 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,043 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e15a8ed[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,060 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,060 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,062 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,062 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,062 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,073 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,074 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:53,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:53,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,174 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,178 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,180 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb67e2d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,198 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676893178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676893180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676893210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676893210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676893210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676893312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676893316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676893316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676893316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676893316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,328 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,331 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1020e68a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,354 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,354 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,365 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,365 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,365 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,365 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,365 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,500 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@943599a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676893516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676893519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676893519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676893521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676893525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,531 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,531 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,533 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,533 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,533 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:53,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,540 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,540 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/fc608e38c1e7479092ce71a96cfd58c2 2023-03-24 16:53:53,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/8a733253a6cd4f56a8e5d3017e9098dc 2023-03-24 16:53:53,640 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,646 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/06d3e9abdef44148a3b6e7b72b2d9848 2023-03-24 16:53:53,654 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/fc608e38c1e7479092ce71a96cfd58c2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2 2023-03-24 16:53:53,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,667 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,667 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,667 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,667 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2, entries=150, sequenceid=584, filesize=11.9 K 2023-03-24 16:53:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/8a733253a6cd4f56a8e5d3017e9098dc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc 2023-03-24 16:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a82da9f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc, entries=150, sequenceid=584, filesize=11.9 K 2023-03-24 16:53:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,680 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,680 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/06d3e9abdef44148a3b6e7b72b2d9848 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848 2023-03-24 16:53:53,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848, entries=150, sequenceid=584, filesize=11.9 K 2023-03-24 16:53:53,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for d3c5b00db9700342afae5fb089181b24 in 536ms, sequenceid=584, compaction requested=true 2023-03-24 16:53:53,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:53,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:53,691 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:53,691 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:53:53,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:53,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,691 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:53,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:53,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,693 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,693 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:53,693 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,693 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5d4d1b8d772b4ecd8b92b6b4934c72c4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.1 K 2023-03-24 16:53:53,694 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:53:53,694 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:53,694 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,694 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7a451b5af9d1494e91e76e4168821595, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.1 K 2023-03-24 16:53:53,694 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5d4d1b8d772b4ecd8b92b6b4934c72c4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679676831349 2023-03-24 16:53:53,696 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7a451b5af9d1494e91e76e4168821595, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679676831349 2023-03-24 16:53:53,706 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 46d345d2d9d142c18671f4732191cac8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679676831391 2023-03-24 16:53:53,707 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting de9e25c468704651baecd5ece7bbdba7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679676831391 2023-03-24 16:53:53,707 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting fc608e38c1e7479092ce71a96cfd58c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:53,707 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8a733253a6cd4f56a8e5d3017e9098dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:53,721 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:53,726 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:53,776 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/d200968bc2784eb092200d9ac97d1a7c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d200968bc2784eb092200d9ac97d1a7c 2023-03-24 16:53:53,783 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into d200968bc2784eb092200d9ac97d1a7c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:53,783 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:53,783 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=13, startTime=1679676833691; duration=0sec 2023-03-24 16:53:53,783 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:53,783 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:53:53,786 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:53:53,786 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:53:53,786 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,786 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b4faddb34a214b92b32a50063a5cf311, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=47.9 K 2023-03-24 16:53:53,787 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b4faddb34a214b92b32a50063a5cf311, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679676830186 2023-03-24 16:53:53,789 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 079149308ead4dec8e72c46949d284e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679676831349 2023-03-24 16:53:53,789 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d35317ba8cff431aa17149c747973891, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679676831391 2023-03-24 16:53:53,789 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 06d3e9abdef44148a3b6e7b72b2d9848, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:53,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,800 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,820 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:53,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:53,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:53,830 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,830 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,830 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,836 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@469988f4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:53,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:53,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676893838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676893840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676893842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676893841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676893842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:53,857 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,857 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:53,859 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:53,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:53,865 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:53,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:53,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:53,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:53,865 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:53,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,871 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,874 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,874 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:53,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:53,875 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:53,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:53,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:53,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:53,884 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/bd45dacf166a4aafa88b6153c86dccc7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/bd45dacf166a4aafa88b6153c86dccc7 2023-03-24 16:53:53,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/2749c0d28ae84eabb83ae6bff09734bf 2023-03-24 16:53:53,895 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into bd45dacf166a4aafa88b6153c86dccc7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:53,895 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:53,895 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=12, startTime=1679676833691; duration=0sec 2023-03-24 16:53:53,895 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:53,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676893950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676893951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676893953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676893958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676893961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:53,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:53,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:53,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:53,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:53,981 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:53,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:53,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:53,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:53,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:53,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:53,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:53,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:53,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:53,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:53,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:53,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:53,999 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519a221b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,005 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,005 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,005 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,005 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,005 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,012 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,012 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,012 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,012 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,012 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,018 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,021 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,021 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,021 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,122 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,132 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:54,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676894153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676894154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:54,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676894157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42862e3b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,161 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,161 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676894161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,163 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676894163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,168 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,169 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,180 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/af6dd1bd5f8b42a3a717eb0c22c60aad as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/af6dd1bd5f8b42a3a717eb0c22c60aad 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:54,200 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into af6dd1bd5f8b42a3a717eb0c22c60aad(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:54,200 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:54,200 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=13, startTime=1679676833691; duration=0sec 2023-03-24 16:53:54,200 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:54,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,287 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,302 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:54,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4edc42d6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/4c5b3dff952342879e8662554d5dbd5d 2023-03-24 16:53:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,331 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,331 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,351 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,352 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,352 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,452 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676894456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676894456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676894461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,464 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676894463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676894467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,479 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,479 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,479 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,479 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,484 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4488add1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,506 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,511 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,511 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:54,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,636 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,638 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,639 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,639 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4482d5a1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,674 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,674 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,693 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/1841b9b11b874cd19830f84e72dc01a8 2023-03-24 16:53:54,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,797 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/2749c0d28ae84eabb83ae6bff09734bf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf 2023-03-24 16:53:54,800 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:54,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf, entries=150, sequenceid=605, filesize=11.9 K 2023-03-24 16:53:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/4c5b3dff952342879e8662554d5dbd5d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d 2023-03-24 16:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:54,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3786ccfc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:54,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:54,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,811 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:54,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:54,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d, entries=150, sequenceid=605, filesize=11.9 K 2023-03-24 16:53:54,817 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/1841b9b11b874cd19830f84e72dc01a8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8 2023-03-24 16:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:54,822 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:54,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:54,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:54,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:54,822 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:54,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8, entries=150, sequenceid=605, filesize=11.9 K 2023-03-24 16:53:54,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d3c5b00db9700342afae5fb089181b24 in 1000ms, sequenceid=605, compaction requested=false 2023-03-24 16:53:54,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:54,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:54,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:54,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:54,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:54,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:54,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:54,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:54,838 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:54,838 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:54,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:54,938 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:54,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:54,949 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:54,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:54,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:54,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:54,953 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:54,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:54,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:54,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:54,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:54,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:54,966 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:54,966 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:54,966 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:54,966 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:54,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:54,966 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 16:53:54,967 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:54,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:54,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:54,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:54,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:54,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:54,990 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/6cfd0886e69240a69ebf82fb72737349 2023-03-24 16:53:55,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676894998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,041 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/37663d6795654d81bc8c72ff77e51124 2023-03-24 16:53:55,075 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/29a816732c7743f496326c0693f791da 2023-03-24 16:53:55,089 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/6cfd0886e69240a69ebf82fb72737349 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349 2023-03-24 16:53:55,097 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349, entries=200, sequenceid=621, filesize=14.3 K 2023-03-24 16:53:55,099 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/37663d6795654d81bc8c72ff77e51124 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124 2023-03-24 16:53:55,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676895104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,108 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124, entries=150, sequenceid=621, filesize=11.9 K 2023-03-24 16:53:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,121 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/29a816732c7743f496326c0693f791da as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da 2023-03-24 16:53:55,127 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da, entries=150, sequenceid=621, filesize=11.9 K 2023-03-24 16:53:55,128 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d3c5b00db9700342afae5fb089181b24 in 162ms, sequenceid=621, compaction requested=true 2023-03-24 16:53:55,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:55,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b60f228[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,139 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,139 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,141 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,141 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,147 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,147 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,150 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,150 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,150 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,151 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,154 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,161 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,161 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,261 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:55,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:55,272 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:55,272 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,272 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:55,272 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:55,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,280 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,282 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,282 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:55,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:55,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:55,283 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,283 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:55,284 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,306 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/9d05f96eaf954f73a79840c8786ac934 2023-03-24 16:53:55,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:55,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:55,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676895333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,343 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/e194f41edcb5459bb5d0ceecba63851f 2023-03-24 16:53:55,358 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/6ba88528c3874aaf9d68ca46f94d0600 2023-03-24 16:53:55,364 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/9d05f96eaf954f73a79840c8786ac934 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934 2023-03-24 16:53:55,370 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934, entries=150, sequenceid=640, filesize=11.9 K 2023-03-24 16:53:55,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/e194f41edcb5459bb5d0ceecba63851f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f 2023-03-24 16:53:55,379 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f, entries=150, sequenceid=640, filesize=11.9 K 2023-03-24 16:53:55,380 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/6ba88528c3874aaf9d68ca46f94d0600 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600 2023-03-24 16:53:55,387 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600, entries=150, sequenceid=640, filesize=11.9 K 2023-03-24 16:53:55,388 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d3c5b00db9700342afae5fb089181b24 in 104ms, sequenceid=640, compaction requested=true 2023-03-24 16:53:55,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:55,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8732b0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,395 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,395 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,395 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,402 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,402 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,406 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,409 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,413 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,413 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:55,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:53:55,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:55,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:55,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:55,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:55,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d836ccdda28b45708a6835cef06c297c 2023-03-24 16:53:55,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676895459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:55,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:55,516 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:55,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:55,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:55,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,519 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:55,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:55,521 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,521 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:55,521 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:55,521 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@487ac39c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,527 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,535 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,535 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,538 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,538 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,538 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,545 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676895564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,646 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:55,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:55,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:55,649 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:55,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:55,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:55,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,651 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,652 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,652 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:55,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:55,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:55,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:55,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499a1743[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,664 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,665 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,674 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676895765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676895766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676895766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676895767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:55,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676895767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:55,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:55,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:55,779 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:55,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:55,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:55,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:55,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:55,785 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:55,785 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,785 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:55,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:55,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:55,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,789 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55774dea[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,798 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,798 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,800 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,807 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,810 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,810 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/99d5fa7d11b1482d94b7a2b32dc7b60d 2023-03-24 16:53:55,910 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:55,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:55,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:55,914 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:55,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:55,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:55,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:55,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:55,919 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,919 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:55,919 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:55,919 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:55,921 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7646f746[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:55,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:55,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:55,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:55,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:55,932 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:55,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:55,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:55,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:55,932 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:55,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:55,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:55,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:55,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:55,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:55,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:55,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:55,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:55,939 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:55,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:55,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:56,040 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:56,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:56,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:56,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:56,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:56,047 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:56,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:56,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:56,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:56,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:56,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:56,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:56,053 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,053 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:56,053 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:56,053 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:56,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683eac31[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:56,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:56,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:56,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:56,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:56,071 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:56,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:56,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:56,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:56,071 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:56,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676896072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676896074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676896074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676896077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676896077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,083 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:56,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:56,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:56,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:56,088 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:56,088 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,088 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:56,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:56,188 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:56,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:56,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:56,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:56,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:56,191 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:56,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:56,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:56,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:56,194 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,194 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:56,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:56,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71079e7a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:56,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:56,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:56,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:56,208 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:56,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:56,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:56,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:56,209 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:56,210 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,210 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:56,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,211 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:56,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:56,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:56,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:56,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:56,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:56,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:56,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/686fa4cc90154ec1bb288583f0dc7d5d 2023-03-24 16:53:56,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d836ccdda28b45708a6835cef06c297c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c 2023-03-24 16:53:56,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c, entries=200, sequenceid=654, filesize=14.3 K 2023-03-24 16:53:56,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/99d5fa7d11b1482d94b7a2b32dc7b60d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d 2023-03-24 16:53:56,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d, entries=150, sequenceid=654, filesize=11.9 K 2023-03-24 16:53:56,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/686fa4cc90154ec1bb288583f0dc7d5d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d 2023-03-24 16:53:56,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d, entries=150, sequenceid=654, filesize=11.9 K 2023-03-24 16:53:56,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d3c5b00db9700342afae5fb089181b24 in 872ms, sequenceid=654, compaction requested=true 2023-03-24 16:53:56,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:56,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:56,307 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:53:56,307 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:53:56,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:56,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:56,309 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66322 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:53:56,309 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:56,309 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:53:56,309 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,309 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:56,309 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/af6dd1bd5f8b42a3a717eb0c22c60aad, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=64.8 K 2023-03-24 16:53:56,309 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,309 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d200968bc2784eb092200d9ac97d1a7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=60.0 K 2023-03-24 16:53:56,310 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting af6dd1bd5f8b42a3a717eb0c22c60aad, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:56,313 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d200968bc2784eb092200d9ac97d1a7c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:56,313 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 2749c0d28ae84eabb83ae6bff09734bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1679676833179 2023-03-24 16:53:56,313 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4c5b3dff952342879e8662554d5dbd5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1679676833179 2023-03-24 16:53:56,313 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6cfd0886e69240a69ebf82fb72737349, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679676833837 2023-03-24 16:53:56,313 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 37663d6795654d81bc8c72ff77e51124, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679676833840 2023-03-24 16:53:56,314 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9d05f96eaf954f73a79840c8786ac934, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679676834989 2023-03-24 16:53:56,314 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e194f41edcb5459bb5d0ceecba63851f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679676834989 2023-03-24 16:53:56,314 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d836ccdda28b45708a6835cef06c297c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:56,314 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 99d5fa7d11b1482d94b7a2b32dc7b60d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:56,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:56,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:56,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:56,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:56,333 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:56,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:56,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:56,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:56,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:56,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:56,340 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:56,340 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#59 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:56,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:56,345 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:56,346 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:56,346 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:56,347 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:56,371 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/b479fb87ba57443fbeefce7c58c50560 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b479fb87ba57443fbeefce7c58c50560 2023-03-24 16:53:56,379 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into b479fb87ba57443fbeefce7c58c50560(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:56,379 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:56,379 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=11, startTime=1679676836307; duration=0sec 2023-03-24 16:53:56,379 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:56,379 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:53:56,383 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:53:56,383 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:53:56,383 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:56,383 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/bd45dacf166a4aafa88b6153c86dccc7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=60.0 K 2023-03-24 16:53:56,383 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting bd45dacf166a4aafa88b6153c86dccc7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679676832528 2023-03-24 16:53:56,384 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1841b9b11b874cd19830f84e72dc01a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1679676833179 2023-03-24 16:53:56,384 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 29a816732c7743f496326c0693f791da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679676833840 2023-03-24 16:53:56,385 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 6ba88528c3874aaf9d68ca46f94d0600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679676834989 2023-03-24 16:53:56,385 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 686fa4cc90154ec1bb288583f0dc7d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:56,419 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#62 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:56,449 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/2fb732bca1794c8c94d4cf5355f8481b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/2fb732bca1794c8c94d4cf5355f8481b 2023-03-24 16:53:56,457 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 2fb732bca1794c8c94d4cf5355f8481b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:56,457 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:56,457 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=11, startTime=1679676836307; duration=0sec 2023-03-24 16:53:56,457 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:56,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:56,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:56,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676896585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676896587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676896589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676896591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676896593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676896694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676896694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676896695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676896696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,756 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/aa2014462ab24deeb1b3f5b0bd0aeba0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/aa2014462ab24deeb1b3f5b0bd0aeba0 2023-03-24 16:53:56,762 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into aa2014462ab24deeb1b3f5b0bd0aeba0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:56,762 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:56,762 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=11, startTime=1679676836307; duration=0sec 2023-03-24 16:53:56,762 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:56,766 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/b784af6dfb814ad1872f1191baa2fda7 2023-03-24 16:53:56,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676896896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676896898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676896899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:56,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676896903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,181 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/64267ede27c4411087de658dca32e7a5 2023-03-24 16:53:57,196 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ca5d68b53f044cc99cc7dc5b8c22dc32 2023-03-24 16:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/b784af6dfb814ad1872f1191baa2fda7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7 2023-03-24 16:53:57,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676897202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676897202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676897202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676897204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,208 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 16:53:57,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/64267ede27c4411087de658dca32e7a5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5 2023-03-24 16:53:57,215 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 16:53:57,216 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ca5d68b53f044cc99cc7dc5b8c22dc32 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32 2023-03-24 16:53:57,222 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 16:53:57,223 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d3c5b00db9700342afae5fb089181b24 in 877ms, sequenceid=673, compaction requested=false 2023-03-24 16:53:57,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:57,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:57,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:57,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:57,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7085ad[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:57,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:57,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:57,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:57,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:57,235 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:57,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:57,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:57,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:57,235 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:57,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:57,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:57,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:57,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:57,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:57,249 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:57,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:57,250 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:57,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:57,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:57,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:57,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,350 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:57,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:57,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:57,353 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:57,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:57,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:57,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:57,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:57,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:57,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:57,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:57,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:57,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:57,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:57,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:57,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:57,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:57,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:57,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:57,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:57,359 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:57,360 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:57,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:57,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:57,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:57,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676897629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676897705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676897705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676897705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676897710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676897732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:57,777 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/a9cca93f0e1d4796bcd609f668fbefbd 2023-03-24 16:53:57,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676897934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,192 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/0870438766b34268b8264f6d65ca5e47 2023-03-24 16:53:58,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676898235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,605 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/4aa33ffa62524ce18aa21c6808f3ef30 2023-03-24 16:53:58,611 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/a9cca93f0e1d4796bcd609f668fbefbd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd 2023-03-24 16:53:58,616 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd, entries=150, sequenceid=690, filesize=11.9 K 2023-03-24 16:53:58,617 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/0870438766b34268b8264f6d65ca5e47 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47 2023-03-24 16:53:58,622 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47, entries=150, sequenceid=690, filesize=11.9 K 2023-03-24 16:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/4aa33ffa62524ce18aa21c6808f3ef30 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30 2023-03-24 16:53:58,628 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30, entries=150, sequenceid=690, filesize=11.9 K 2023-03-24 16:53:58,628 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d3c5b00db9700342afae5fb089181b24 in 1268ms, sequenceid=690, compaction requested=true 2023-03-24 16:53:58,629 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:58,629 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:58,630 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f4dade2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:58,632 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:58,632 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:58,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:58,640 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:58,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:58,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:58,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:58,640 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:58,643 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,643 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,643 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:58,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:58,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:58,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:58,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:58,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:58,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:58,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:58,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:58,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:58,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d4515bb6b1d64c5091139e4baace4550 2023-03-24 16:53:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676898725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676898727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676898728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676898729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676898739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:58,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:58,752 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:58,754 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:58,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:58,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:58,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:58,760 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@420fdd1e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:58,760 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,763 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:58,763 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:58,763 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:58,763 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:58,769 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:58,772 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:58,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:58,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:58,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:58,772 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:58,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:58,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:58,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:58,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:58,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:58,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,779 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,779 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,779 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:58,779 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,779 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:58,779 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,779 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676898830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676898831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676898831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676898832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,879 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:58,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:58,892 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:58,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:58,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:58,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:58,893 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,893 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:58,895 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:58,897 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,897 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:58,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:58,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:58,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:58,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:58,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:58,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:58,906 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:58,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f366f6b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:58,910 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:58,910 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:58,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:58,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:58,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:58,917 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:58,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:58,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:58,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:58,918 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:58,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:58,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:58,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:58,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:58,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:58,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:58,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:58,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:58,928 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:58,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:58,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:58,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:58,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:58,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:58,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:59,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:59,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:59,033 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:59,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:59,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:59,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676899033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676899033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676899034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676899034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,035 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,036 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,036 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:59,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:59,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:59,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:59,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:59,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1571cecd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:59,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:59,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:59,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:59,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:59,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:59,046 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:59,048 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:59,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:59,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:59,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:59,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:59,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:59,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:59,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:59,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:59,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/aa40a82f47b14e48834af2f827ed521b 2023-03-24 16:53:59,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:59,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:59,158 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:59,158 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,158 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:59,158 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:59,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:59,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:59,163 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:59,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:59,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:59,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:59,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2761433c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:59,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:59,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,169 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:59,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:59,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:59,177 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:59,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:59,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:59,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:59,177 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:59,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:59,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:59,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:59,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:59,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:59,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:59,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:59,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:59,304 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:59,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:59,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:59,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,306 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:59,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:59,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:59,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:59,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:59,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:59,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:59,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bba9de3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:59,324 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:59,324 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,325 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:59,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:59,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:59,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:59,327 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:59,331 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:59,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:59,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:59,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:59,331 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:59,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676899337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676899337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676899340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676899340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:59,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:59,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:59,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:59,347 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,447 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:59,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:59,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:59,451 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:59,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:59,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:59,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:59,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:59,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:59,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:59,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:59,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:59,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:59,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:59,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:53:59,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:53:59,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27cbf4c8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:53:59,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,475 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:53:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:53:59,475 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,476 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:53:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:53:59,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:53:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:53:59,480 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:53:59,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:53:59,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:53:59,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:53:59,480 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:53:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:53:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,483 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:53:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:53:59,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:53:59,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:53:59,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:53:59,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:53:59,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:53:59,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:53:59,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:53:59,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/8735f484a9654d1894d7f2b87740ea71 2023-03-24 16:53:59,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/d4515bb6b1d64c5091139e4baace4550 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550 2023-03-24 16:53:59,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550, entries=150, sequenceid=709, filesize=11.9 K 2023-03-24 16:53:59,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/aa40a82f47b14e48834af2f827ed521b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b 2023-03-24 16:53:59,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b, entries=150, sequenceid=709, filesize=11.9 K 2023-03-24 16:53:59,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/8735f484a9654d1894d7f2b87740ea71 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71 2023-03-24 16:53:59,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71, entries=150, sequenceid=709, filesize=11.9 K 2023-03-24 16:53:59,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d3c5b00db9700342afae5fb089181b24 in 870ms, sequenceid=709, compaction requested=true 2023-03-24 16:53:59,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:59,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:59,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:59,580 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:53:59,580 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:53:59,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:59,582 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:53:59,582 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:53:59,582 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:53:59,582 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,582 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b479fb87ba57443fbeefce7c58c50560, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=48.3 K 2023-03-24 16:53:59,582 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:53:59,583 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,583 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/aa2014462ab24deeb1b3f5b0bd0aeba0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=48.3 K 2023-03-24 16:53:59,583 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b479fb87ba57443fbeefce7c58c50560, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:59,583 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting aa2014462ab24deeb1b3f5b0bd0aeba0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:59,584 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 64267ede27c4411087de658dca32e7a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679676835448 2023-03-24 16:53:59,584 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b784af6dfb814ad1872f1191baa2fda7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679676835448 2023-03-24 16:53:59,584 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 0870438766b34268b8264f6d65ca5e47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679676836588 2023-03-24 16:53:59,584 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a9cca93f0e1d4796bcd609f668fbefbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679676836588 2023-03-24 16:53:59,585 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting aa40a82f47b14e48834af2f827ed521b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:53:59,585 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d4515bb6b1d64c5091139e4baace4550, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:53:59,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:53:59,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:53:59,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:53:59,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:53:59,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:53:59,593 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:53:59,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:53:59,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:53:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:53:59,597 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:59,597 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:59,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:53:59,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:53:59,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:53:59,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:53:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:53:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:53:59,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:53:59,604 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:53:59,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:53:59,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:59,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:53:59,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:59,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:53:59,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:53:59,628 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/7668c6aa3dbb44f1a70f4819b96deb0d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7668c6aa3dbb44f1a70f4819b96deb0d 2023-03-24 16:53:59,630 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/0050dbdc12604f9a9a3da5945ff387b3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0050dbdc12604f9a9a3da5945ff387b3 2023-03-24 16:53:59,637 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into 7668c6aa3dbb44f1a70f4819b96deb0d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:59,637 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:59,637 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=12, startTime=1679676839580; duration=0sec 2023-03-24 16:53:59,638 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:53:59,638 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:53:59,638 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/1d579dbe238944e7ab2c1b6b7b5a8743 2023-03-24 16:53:59,639 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into 0050dbdc12604f9a9a3da5945ff387b3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:59,639 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:59,639 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=12, startTime=1679676839580; duration=0sec 2023-03-24 16:53:59,639 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:59,639 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:53:59,639 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:53:59,640 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:53:59,640 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/2fb732bca1794c8c94d4cf5355f8481b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=48.2 K 2023-03-24 16:53:59,640 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2fb732bca1794c8c94d4cf5355f8481b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679676835323 2023-03-24 16:53:59,641 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ca5d68b53f044cc99cc7dc5b8c22dc32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679676835448 2023-03-24 16:53:59,641 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4aa33ffa62524ce18aa21c6808f3ef30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679676836588 2023-03-24 16:53:59,642 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8735f484a9654d1894d7f2b87740ea71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:53:59,656 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:53:59,664 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/1f1b19669bc34cbc9e061d3da5ca91c9 2023-03-24 16:53:59,673 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/1b91286e693741d89ada49188a78b3c9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1b91286e693741d89ada49188a78b3c9 2023-03-24 16:53:59,681 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 1b91286e693741d89ada49188a78b3c9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:53:59,681 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:53:59,681 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=12, startTime=1679676839580; duration=0sec 2023-03-24 16:53:59,681 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:53:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:53:59,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:53:59,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676899771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676899840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676899841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676899845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676899846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:53:59,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:53:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676899873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676900076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,084 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ad164c225aaa4851b3eaf3268cb977a4 2023-03-24 16:54:00,092 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/1d579dbe238944e7ab2c1b6b7b5a8743 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743 2023-03-24 16:54:00,097 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743, entries=150, sequenceid=723, filesize=11.9 K 2023-03-24 16:54:00,098 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/1f1b19669bc34cbc9e061d3da5ca91c9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9 2023-03-24 16:54:00,104 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9, entries=150, sequenceid=723, filesize=11.9 K 2023-03-24 16:54:00,105 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ad164c225aaa4851b3eaf3268cb977a4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4 2023-03-24 16:54:00,110 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4, entries=150, sequenceid=723, filesize=11.9 K 2023-03-24 16:54:00,111 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d3c5b00db9700342afae5fb089181b24 in 507ms, sequenceid=723, compaction requested=false 2023-03-24 16:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,113 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd7415a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,118 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,119 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,120 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,123 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,123 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,125 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,136 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,136 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,236 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,240 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,242 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,244 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:00,245 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,253 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/ce95a4719fc14408a93376f04e514efa 2023-03-24 16:54:00,268 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/ee0abaaca8eb4ff2b4488a95703a43ed 2023-03-24 16:54:00,280 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/84ecbc6d58e04a5db658ecf3f16d7e5c 2023-03-24 16:54:00,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/ce95a4719fc14408a93376f04e514efa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa 2023-03-24 16:54:00,297 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa, entries=150, sequenceid=744, filesize=11.9 K 2023-03-24 16:54:00,298 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/ee0abaaca8eb4ff2b4488a95703a43ed as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed 2023-03-24 16:54:00,308 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed, entries=150, sequenceid=744, filesize=11.9 K 2023-03-24 16:54:00,309 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/84ecbc6d58e04a5db658ecf3f16d7e5c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c 2023-03-24 16:54:00,314 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c, entries=150, sequenceid=744, filesize=11.9 K 2023-03-24 16:54:00,315 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for d3c5b00db9700342afae5fb089181b24 in 71ms, sequenceid=744, compaction requested=true 2023-03-24 16:54:00,315 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:00,315 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,317 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11028d5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,326 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,326 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,328 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,328 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,328 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,334 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,334 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:00,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:00,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:00,435 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,439 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676840401, max timestamp=1679676840442Num uniques 859; 2023-03-24 16:54:00,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676840401, max timestamp=1679676840442Num uniques 859; 2023-03-24 16:54:00,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:00,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676840401, max timestamp=1679676840442Num uniques 859; 2023-03-24 16:54:00,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:00,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:00,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:00,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,459 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e185f19[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,470 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,470 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,471 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,471 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,472 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,472 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,476 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,477 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,477 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676900485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,577 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,580 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:00,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@268d8da6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676900588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,590 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,591 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,595 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,595 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,600 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,602 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,603 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,603 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:00,603 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,703 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,707 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:00,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2056d5bc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,725 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,725 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,725 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,725 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,725 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,728 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,729 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,732 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,732 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,733 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,733 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676900792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/dd136d1527f54b4a84726cac5cff7ae6 2023-03-24 16:54:00,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,836 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,840 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,841 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,842 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,842 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,842 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:00,842 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,846 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676900846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@253bdc9c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676900847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,854 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,854 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676900855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,857 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,858 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676900857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:00,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:00,863 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,863 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:00,863 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,863 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:00,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:00,977 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:00,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:00,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:00,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,979 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:00,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:00,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,980 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:00,981 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,981 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:00,981 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:00,981 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:00,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:00,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3947bfb5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:00,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:00,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:00,989 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:00,993 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:00,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:00,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:00,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:00,993 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:00,995 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,995 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:00,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,995 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:00,995 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:00,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,997 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:01,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:01,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676901096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,106 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,108 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,110 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,110 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,110 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:01,110 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:01,111 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736de586[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:01,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:01,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,121 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:01,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:01,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:01,125 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:01,125 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:01,125 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:01,125 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:01,125 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:01,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:01,131 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/1532556eea4e4505ac9a24e129ff6e8e 2023-03-24 16:54:01,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,234 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,237 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:01,241 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:01,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65803562[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:01,242 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:01,244 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,244 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:01,247 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:01,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:01,250 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:01,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:01,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:01,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:01,250 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:01,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,258 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,364 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,366 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:01,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,372 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e24ca72[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:01,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:01,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:01,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:01,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:01,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:01,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:01,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:01,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:01,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:01,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:01,386 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,386 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,386 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,390 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,394 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,394 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:01,494 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,497 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:01,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:01,505 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:01,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a9a7414[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:01,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:01,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:01,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:01,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:01,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:01,514 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:01,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:01,519 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:01,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:01,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:01,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:01,519 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:01,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,527 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:01,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676901599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/748ce96b39014e9d84ba90c29aece371 2023-03-24 16:54:01,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,631 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/dd136d1527f54b4a84726cac5cff7ae6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6 2023-03-24 16:54:01,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6, entries=250, sequenceid=754, filesize=16.7 K 2023-03-24 16:54:01,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/1532556eea4e4505ac9a24e129ff6e8e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:01,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:01,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647f314[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:01,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:01,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:01,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:01,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:01,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,651 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:01,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e, entries=150, sequenceid=754, filesize=11.9 K 2023-03-24 16:54:01,652 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:01,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/748ce96b39014e9d84ba90c29aece371 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371 2023-03-24 16:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:01,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:01,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:01,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:01,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:01,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:01,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:01,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371, entries=150, sequenceid=754, filesize=11.9 K 2023-03-24 16:54:01,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:01,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,661 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d3c5b00db9700342afae5fb089181b24 in 1269ms, sequenceid=754, compaction requested=true 2023-03-24 16:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:01,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:01,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:01,661 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:01,661 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:01,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:01,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:01,663 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54442 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:01,663 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:54:01,663 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,663 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0050dbdc12604f9a9a3da5945ff387b3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=53.2 K 2023-03-24 16:54:01,664 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0050dbdc12604f9a9a3da5945ff387b3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:54:01,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:01,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:54:01,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,664 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1d579dbe238944e7ab2c1b6b7b5a8743, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679676838724 2023-03-24 16:54:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:01,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7668c6aa3dbb44f1a70f4819b96deb0d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=48.4 K 2023-03-24 16:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,665 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ce95a4719fc14408a93376f04e514efa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1679676839758 2023-03-24 16:54:01,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7668c6aa3dbb44f1a70f4819b96deb0d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:54:01,665 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting dd136d1527f54b4a84726cac5cff7ae6, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:01,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1f1b19669bc34cbc9e061d3da5ca91c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679676838724 2023-03-24 16:54:01,666 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ee0abaaca8eb4ff2b4488a95703a43ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1679676839758 2023-03-24 16:54:01,667 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1532556eea4e4505ac9a24e129ff6e8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:01,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:01,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:01,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:01,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:01,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,667 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:01,675 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:01,676 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:01,768 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:01,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:01,771 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:01,771 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:01,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:01,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,772 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:01,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:01,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:01,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:01,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:01,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:01,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:01,780 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:01,780 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:01,781 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:01,781 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:01,797 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/86d575950dd84819a1751e232fbc3af5 2023-03-24 16:54:02,090 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/9158f5e430f148d0b0f42ad16a9342e6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9158f5e430f148d0b0f42ad16a9342e6 2023-03-24 16:54:02,096 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/8eae789c7d634cf4ad5e3609b5ea82cd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8eae789c7d634cf4ad5e3609b5ea82cd 2023-03-24 16:54:02,106 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into 9158f5e430f148d0b0f42ad16a9342e6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:02,106 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:02,106 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=12, startTime=1679676841661; duration=0sec 2023-03-24 16:54:02,106 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into 8eae789c7d634cf4ad5e3609b5ea82cd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:02,106 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:02,106 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:02,106 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:02,106 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=12, startTime=1679676841661; duration=0sec 2023-03-24 16:54:02,106 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:02,108 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:02,108 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:54:02,108 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,108 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1b91286e693741d89ada49188a78b3c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=48.4 K 2023-03-24 16:54:02,109 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1b91286e693741d89ada49188a78b3c9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679676837609 2023-03-24 16:54:02,110 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ad164c225aaa4851b3eaf3268cb977a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679676838724 2023-03-24 16:54:02,110 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 84ecbc6d58e04a5db658ecf3f16d7e5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1679676839758 2023-03-24 16:54:02,111 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 748ce96b39014e9d84ba90c29aece371, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:02,127 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:02,209 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/d7cee85e454c4346b392e7b4cf63e8bf 2023-03-24 16:54:02,545 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/854c153151974d38ab8fdbaba03a52d3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/854c153151974d38ab8fdbaba03a52d3 2023-03-24 16:54:02,553 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 854c153151974d38ab8fdbaba03a52d3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:02,553 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:02,553 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=12, startTime=1679676841661; duration=0sec 2023-03-24 16:54:02,553 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:02,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:02,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:02,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676902628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,632 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/e2c9dfca252a46f588ff7b58a8c6a973 2023-03-24 16:54:02,639 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/86d575950dd84819a1751e232fbc3af5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5 2023-03-24 16:54:02,645 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5, entries=150, sequenceid=790, filesize=11.9 K 2023-03-24 16:54:02,646 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/d7cee85e454c4346b392e7b4cf63e8bf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf 2023-03-24 16:54:02,652 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf, entries=150, sequenceid=790, filesize=11.9 K 2023-03-24 16:54:02,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/e2c9dfca252a46f588ff7b58a8c6a973 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973 2023-03-24 16:54:02,659 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973, entries=150, sequenceid=790, filesize=11.9 K 2023-03-24 16:54:02,660 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for d3c5b00db9700342afae5fb089181b24 in 880ms, sequenceid=790, compaction requested=false 2023-03-24 16:54:02,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:02,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:02,663 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6288fd9b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:02,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:02,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:02,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:02,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:02,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:02,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:02,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:02,674 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:02,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:02,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:02,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:02,674 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:02,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:02,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:02,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:02,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:02,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:02,682 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:02,683 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:02,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:02,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:02,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:54:02,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:02,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:02,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:02,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:02,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:02,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:02,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/bca3aa4b70304f8491cdbd60a7212bcc 2023-03-24 16:54:02,783 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:02,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:02,800 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:02,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:02,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:02,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:02,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:02,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:02,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:02,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:02,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676842768, max timestamp=1679676842810Num uniques 859; 2023-03-24 16:54:02,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:02,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676842782, max timestamp=1679676842810Num uniques 859; 2023-03-24 16:54:02,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:02,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:02,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:02,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:02,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:02,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676842782, max timestamp=1679676842810Num uniques 859; 2023-03-24 16:54:02,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:02,819 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:02,822 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a47cce2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:02,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:02,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,830 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:02,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:02,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:02,835 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:02,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:02,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:02,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:02,835 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:02,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:02,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:02,838 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:02,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:02,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:02,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:02,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:02,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:02,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676902867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50860 deadline: 1679676902867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676902868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50830 deadline: 1679676902870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,874 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:02,877 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:02,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50842 deadline: 1679676902883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,894 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:02,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:02,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:02,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:02,958 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:02,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:02,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:02,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:02,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,960 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,960 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:02,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:02,960 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,960 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:02,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:02,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:02,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:02,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:02,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:02,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676902972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:02,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:02,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:02,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:02,979 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:02,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676902978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1334296a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:02,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:02,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:02,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:02,995 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:03,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:03,000 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:03,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:03,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:03,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:03,000 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:03,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,008 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:03,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:03,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:03,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:03,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,011 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:03,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:03,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:03,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,012 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,012 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,012 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,012 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:03,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:03,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:03,115 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:03,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:03,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:03,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:03,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:03,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:03,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:03,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:03,119 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,119 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:03,119 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:03,119 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:03,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad2a74b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:03,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:03,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,126 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:03,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:03,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:03,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:03,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:03,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:03,130 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:03,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:03,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:03,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:03,130 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:03,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:03,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:03,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:03,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:03,138 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:03,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:03,139 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,139 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,139 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,139 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,139 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,144 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:03,144 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:03,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/f0ba0c86d06141c4bf90d908197b7938 2023-03-24 16:54:03,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676903176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676903182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/039168e3c99048fb822cb51d384863c4 2023-03-24 16:54:03,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/bca3aa4b70304f8491cdbd60a7212bcc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc 2023-03-24 16:54:03,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc, entries=100, sequenceid=804, filesize=9.5 K 2023-03-24 16:54:03,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/f0ba0c86d06141c4bf90d908197b7938 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938 2023-03-24 16:54:03,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938, entries=100, sequenceid=804, filesize=9.5 K 2023-03-24 16:54:03,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/039168e3c99048fb822cb51d384863c4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4 2023-03-24 16:54:03,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4, entries=100, sequenceid=804, filesize=9.5 K 2023-03-24 16:54:03,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for d3c5b00db9700342afae5fb089181b24 in 497ms, sequenceid=804, compaction requested=true 2023-03-24 16:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:03,229 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:03,229 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:03,230 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:03,230 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:54:03,231 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,231 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9158f5e430f148d0b0f42ad16a9342e6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=34.2 K 2023-03-24 16:54:03,231 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:03,231 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:54:03,231 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,231 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9158f5e430f148d0b0f42ad16a9342e6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:03,231 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8eae789c7d634cf4ad5e3609b5ea82cd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=34.2 K 2023-03-24 16:54:03,232 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 86d575950dd84819a1751e232fbc3af5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1679676840471 2023-03-24 16:54:03,232 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8eae789c7d634cf4ad5e3609b5ea82cd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:03,232 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting bca3aa4b70304f8491cdbd60a7212bcc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676842627 2023-03-24 16:54:03,233 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d7cee85e454c4346b392e7b4cf63e8bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1679676840471 2023-03-24 16:54:03,233 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f0ba0c86d06141c4bf90d908197b7938, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676842627 2023-03-24 16:54:03,241 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:03,244 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:03,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:03,249 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:03,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:03,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:03,251 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:03,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:03,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:03,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:03,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:03,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:03,255 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,255 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:03,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:03,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:03,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:03,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:03,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:03,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:03,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:03,272 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,272 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. started... 2023-03-24 16:54:03,272 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-03-24 16:54:03,272 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:03,273 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:03,273 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:03,273 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:03,273 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:03,273 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:03,290 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/2a881232ef6a46a59f2d31344a12a923 2023-03-24 16:54:03,305 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/b64fac9878b34b11a98e100929ad9326 2023-03-24 16:54:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:03,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. as already flushing 2023-03-24 16:54:03,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676903498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676903500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,558 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:62195 2023-03-24 16:54:03,558 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,559 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:62195 2023-03-24 16:54:03,559 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,563 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:62195 2023-03-24 16:54:03,563 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,564 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:62195 2023-03-24 16:54:03,564 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1679676903602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:03,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50824 deadline: 1679676903605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,675 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/a07ac9971de645f4b5548d9333e2b46e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a07ac9971de645f4b5548d9333e2b46e 2023-03-24 16:54:03,680 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into a07ac9971de645f4b5548d9333e2b46e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:03,680 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:03,680 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=13, startTime=1679676843229; duration=0sec 2023-03-24 16:54:03,680 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:03,681 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:03,698 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:03,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:54:03,699 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,699 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/854c153151974d38ab8fdbaba03a52d3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=34.2 K 2023-03-24 16:54:03,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 854c153151974d38ab8fdbaba03a52d3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1679676840383 2023-03-24 16:54:03,700 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e2c9dfca252a46f588ff7b58a8c6a973, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1679676840471 2023-03-24 16:54:03,700 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 039168e3c99048fb822cb51d384863c4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676842627 2023-03-24 16:54:03,700 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/b136f083fdca4a64b6f4f013ace03ebd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b136f083fdca4a64b6f4f013ace03ebd 2023-03-24 16:54:03,706 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into b136f083fdca4a64b6f4f013ace03ebd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:03,706 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:03,707 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=13, startTime=1679676843229; duration=0sec 2023-03-24 16:54:03,707 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:03,707 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:03,746 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/014abac2a7c7447b930e83736b35f73f 2023-03-24 16:54:03,752 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/2a881232ef6a46a59f2d31344a12a923 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923 2023-03-24 16:54:03,757 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 16:54:03,758 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/b64fac9878b34b11a98e100929ad9326 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326 2023-03-24 16:54:03,764 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 16:54:03,765 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/014abac2a7c7447b930e83736b35f73f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f 2023-03-24 16:54:03,769 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 16:54:03,770 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for d3c5b00db9700342afae5fb089181b24 in 498ms, sequenceid=840, compaction requested=false 2023-03-24 16:54:03,770 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:03,770 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:03,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,773 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,773 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:03,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f13e3af[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:03,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:03,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:03,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:03,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:03,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:03,780 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:03,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:03,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:03,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:03,780 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:03,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:03,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:03,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,786 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:03,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:03,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:03,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:03,790 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:03,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:03,790 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:03,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:03,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:03,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:03,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:54:03,805 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:62195 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:03,805 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:03,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:03,809 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:62195 2023-03-24 16:54:03,809 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:03,890 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:04,117 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/16abc890d748484a9399330fa348b42f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/16abc890d748484a9399330fa348b42f 2023-03-24 16:54:04,123 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 16abc890d748484a9399330fa348b42f(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:04,123 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:04,123 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=13, startTime=1679676843229; duration=0sec 2023-03-24 16:54:04,123 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:04,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/5786aeee196548c2ba925d69ca4aa368 2023-03-24 16:54:04,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/8351d1006bee41eba8e4961beacbf070 2023-03-24 16:54:05,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ef6ae4ff41164e76a006354dade25b35 2023-03-24 16:54:05,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/5786aeee196548c2ba925d69ca4aa368 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368 2023-03-24 16:54:05,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368, entries=150, sequenceid=854, filesize=11.9 K 2023-03-24 16:54:05,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/8351d1006bee41eba8e4961beacbf070 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070 2023-03-24 16:54:05,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070, entries=150, sequenceid=854, filesize=11.9 K 2023-03-24 16:54:05,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/ef6ae4ff41164e76a006354dade25b35 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35 2023-03-24 16:54:05,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35, entries=150, sequenceid=854, filesize=11.9 K 2023-03-24 16:54:05,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=6.71 KB/6870 for d3c5b00db9700342afae5fb089181b24 in 1252ms, sequenceid=854, compaction requested=true 2023-03-24 16:54:05,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:05,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:05,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:05,056 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:05,056 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:05,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:05,058 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:05,058 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/A is initiating minor compaction (all files) 2023-03-24 16:54:05,058 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:05,058 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/A in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:05,058 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/B is initiating minor compaction (all files) 2023-03-24 16:54:05,058 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a07ac9971de645f4b5548d9333e2b46e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.7 K 2023-03-24 16:54:05,058 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/B in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:05,058 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b136f083fdca4a64b6f4f013ace03ebd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.7 K 2023-03-24 16:54:05,058 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a07ac9971de645f4b5548d9333e2b46e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676840471 2023-03-24 16:54:05,059 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b136f083fdca4a64b6f4f013ace03ebd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676840471 2023-03-24 16:54:05,059 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 2a881232ef6a46a59f2d31344a12a923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679676842841 2023-03-24 16:54:05,059 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b64fac9878b34b11a98e100929ad9326, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679676842841 2023-03-24 16:54:05,059 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5786aeee196548c2ba925d69ca4aa368, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679676843493 2023-03-24 16:54:05,059 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8351d1006bee41eba8e4961beacbf070, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679676843493 2023-03-24 16:54:05,068 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#A#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:05,069 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#B#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:05,482 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/c4cc96715d51487989f06f0aa633d1b9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/c4cc96715d51487989f06f0aa633d1b9 2023-03-24 16:54:05,482 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/047c618927a34f23a83cb09db9e2fdcd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/047c618927a34f23a83cb09db9e2fdcd 2023-03-24 16:54:05,487 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/B of d3c5b00db9700342afae5fb089181b24 into c4cc96715d51487989f06f0aa633d1b9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:05,487 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:05,487 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/A of d3c5b00db9700342afae5fb089181b24 into 047c618927a34f23a83cb09db9e2fdcd(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:05,488 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/B, priority=13, startTime=1679676845056; duration=0sec 2023-03-24 16:54:05,488 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:05,488 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/A, priority=13, startTime=1679676845056; duration=0sec 2023-03-24 16:54:05,488 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:05,488 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:05,488 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:05,489 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:05,489 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): d3c5b00db9700342afae5fb089181b24/C is initiating minor compaction (all files) 2023-03-24 16:54:05,489 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3c5b00db9700342afae5fb089181b24/C in TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:05,489 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/16abc890d748484a9399330fa348b42f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp, totalSize=36.7 K 2023-03-24 16:54:05,490 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 16abc890d748484a9399330fa348b42f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1679676840471 2023-03-24 16:54:05,490 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 014abac2a7c7447b930e83736b35f73f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679676842841 2023-03-24 16:54:05,491 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ef6ae4ff41164e76a006354dade25b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679676843493 2023-03-24 16:54:05,498 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): d3c5b00db9700342afae5fb089181b24#C#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:05,908 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/4e2cb102f13343979cfa7cc763e56534 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4e2cb102f13343979cfa7cc763e56534 2023-03-24 16:54:05,913 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3c5b00db9700342afae5fb089181b24/C of d3c5b00db9700342afae5fb089181b24 into 4e2cb102f13343979cfa7cc763e56534(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:05,913 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:05,913 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24., storeName=d3c5b00db9700342afae5fb089181b24/C, priority=13, startTime=1679676845056; duration=0sec 2023-03-24 16:54:05,913 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:06,880 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:62195 2023-03-24 16:54:06,880 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:06,908 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df095a to 127.0.0.1:62195 2023-03-24 16:54:06,908 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:06,915 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:62195 2023-03-24 16:54:06,915 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 241 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4680 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4599 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 2060 2023-03-24 16:54:06,916 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 6179 rows 2023-03-24 16:54:06,917 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 2054 2023-03-24 16:54:06,917 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 6158 rows 2023-03-24 16:54:06,917 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:54:06,917 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d61f291 to 127.0.0.1:62195 2023-03-24 16:54:06,917 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:06,923 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:54:06,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:54:06,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:06,939 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676846939"}]},"ts":"1679676846939"} 2023-03-24 16:54:06,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:06,940 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:54:06,942 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:54:06,943 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, UNASSIGN}] 2023-03-24 16:54:06,944 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, UNASSIGN 2023-03-24 16:54:06,945 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d3c5b00db9700342afae5fb089181b24, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:06,945 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676846945"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676846945"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676846945"}]},"ts":"1679676846945"} 2023-03-24 16:54:06,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:07,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:07,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d3c5b00db9700342afae5fb089181b24, disabling compactions & flushes 2023-03-24 16:54:07,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. after waiting 0 ms 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:07,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d3c5b00db9700342afae5fb089181b24 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=A 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=B 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3c5b00db9700342afae5fb089181b24, store=C 2023-03-24 16:54:07,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:07,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:07,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/8177d0607c5e447097cd51be8248d1d0 2023-03-24 16:54:07,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:07,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/d0e13b29979f47f783ad0f36064b224b 2023-03-24 16:54:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:08,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/b64cbc53cef3402bb927d8271950a10c 2023-03-24 16:54:08,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/A/8177d0607c5e447097cd51be8248d1d0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8177d0607c5e447097cd51be8248d1d0 2023-03-24 16:54:08,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8177d0607c5e447097cd51be8248d1d0, entries=100, sequenceid=865, filesize=9.5 K 2023-03-24 16:54:08,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/B/d0e13b29979f47f783ad0f36064b224b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d0e13b29979f47f783ad0f36064b224b 2023-03-24 16:54:08,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d0e13b29979f47f783ad0f36064b224b, entries=100, sequenceid=865, filesize=9.5 K 2023-03-24 16:54:08,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/.tmp/C/b64cbc53cef3402bb927d8271950a10c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b64cbc53cef3402bb927d8271950a10c 2023-03-24 16:54:08,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b64cbc53cef3402bb927d8271950a10c, entries=100, sequenceid=865, filesize=9.5 K 2023-03-24 16:54:08,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d3c5b00db9700342afae5fb089181b24 in 1251ms, sequenceid=865, compaction requested=false 2023-03-24 16:54:08,378 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fb4feee842994f2da7d748e54d32aace, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86f36741a83844dbaefb8854e04eb0fd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5d4d1b8d772b4ecd8b92b6b4934c72c4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/af6dd1bd5f8b42a3a717eb0c22c60aad, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/aa2014462ab24deeb1b3f5b0bd0aeba0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0050dbdc12604f9a9a3da5945ff387b3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9158f5e430f148d0b0f42ad16a9342e6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a07ac9971de645f4b5548d9333e2b46e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368] to archive 2023-03-24 16:54:08,379 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:08,384 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5990d0f43e424e959a4a0b1d32a969e2 2023-03-24 16:54:08,385 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d206de5a7fbf47d9a474b1327ba1713a 2023-03-24 16:54:08,386 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8c693ac5651545a4a285458485a02b4f 2023-03-24 16:54:08,387 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fb4feee842994f2da7d748e54d32aace to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fb4feee842994f2da7d748e54d32aace 2023-03-24 16:54:08,389 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/81defd914d9048acb53a38062f1fce4f 2023-03-24 16:54:08,390 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1f7d727936b84b0aa094efe9d45d5b3c 2023-03-24 16:54:08,391 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/4555bf79d9d24b1c80de5203f8717268 2023-03-24 16:54:08,392 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/675a104d462c47a4a87592ca5f569237 2023-03-24 16:54:08,393 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/45ecad3f69814ea0b622ecedd853c0a7 2023-03-24 16:54:08,394 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86f36741a83844dbaefb8854e04eb0fd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86f36741a83844dbaefb8854e04eb0fd 2023-03-24 16:54:08,396 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/97e1f655590f4611bb74a49f390a1898 2023-03-24 16:54:08,397 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0b73ec470292401ca1488a851a3febd4 2023-03-24 16:54:08,398 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5d4d1b8d772b4ecd8b92b6b4934c72c4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5d4d1b8d772b4ecd8b92b6b4934c72c4 2023-03-24 16:54:08,399 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/46d345d2d9d142c18671f4732191cac8 2023-03-24 16:54:08,401 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/af6dd1bd5f8b42a3a717eb0c22c60aad to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/af6dd1bd5f8b42a3a717eb0c22c60aad 2023-03-24 16:54:08,402 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/fc608e38c1e7479092ce71a96cfd58c2 2023-03-24 16:54:08,403 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2749c0d28ae84eabb83ae6bff09734bf 2023-03-24 16:54:08,404 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/6cfd0886e69240a69ebf82fb72737349 2023-03-24 16:54:08,405 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9d05f96eaf954f73a79840c8786ac934 2023-03-24 16:54:08,406 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d836ccdda28b45708a6835cef06c297c 2023-03-24 16:54:08,407 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/aa2014462ab24deeb1b3f5b0bd0aeba0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/aa2014462ab24deeb1b3f5b0bd0aeba0 2023-03-24 16:54:08,408 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/b784af6dfb814ad1872f1191baa2fda7 2023-03-24 16:54:08,410 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a9cca93f0e1d4796bcd609f668fbefbd 2023-03-24 16:54:08,411 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0050dbdc12604f9a9a3da5945ff387b3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/0050dbdc12604f9a9a3da5945ff387b3 2023-03-24 16:54:08,412 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/d4515bb6b1d64c5091139e4baace4550 2023-03-24 16:54:08,413 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/1d579dbe238944e7ab2c1b6b7b5a8743 2023-03-24 16:54:08,414 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/ce95a4719fc14408a93376f04e514efa 2023-03-24 16:54:08,415 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/dd136d1527f54b4a84726cac5cff7ae6 2023-03-24 16:54:08,417 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9158f5e430f148d0b0f42ad16a9342e6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/9158f5e430f148d0b0f42ad16a9342e6 2023-03-24 16:54:08,418 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/86d575950dd84819a1751e232fbc3af5 2023-03-24 16:54:08,419 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a07ac9971de645f4b5548d9333e2b46e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/a07ac9971de645f4b5548d9333e2b46e 2023-03-24 16:54:08,420 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/bca3aa4b70304f8491cdbd60a7212bcc 2023-03-24 16:54:08,421 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/2a881232ef6a46a59f2d31344a12a923 2023-03-24 16:54:08,423 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/5786aeee196548c2ba925d69ca4aa368 2023-03-24 16:54:08,449 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fc382a2524d446fc8387c3943d9a833f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/659ecd47a2af4f909ce886345dfa05ba, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7a451b5af9d1494e91e76e4168821595, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d200968bc2784eb092200d9ac97d1a7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b479fb87ba57443fbeefce7c58c50560, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7668c6aa3dbb44f1a70f4819b96deb0d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8eae789c7d634cf4ad5e3609b5ea82cd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b136f083fdca4a64b6f4f013ace03ebd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070] to archive 2023-03-24 16:54:08,450 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:08,452 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fb66407c8afd4217bed176deac17a274 2023-03-24 16:54:08,453 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c3d5f766b464206a8863a2e1d8fabe3 2023-03-24 16:54:08,455 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fc382a2524d446fc8387c3943d9a833f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/fc382a2524d446fc8387c3943d9a833f 2023-03-24 16:54:08,456 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/77dd81a5aad74756891e6c67e02d3e3a 2023-03-24 16:54:08,457 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/453e31cc1db44f82b37fcd1f83f0d29e 2023-03-24 16:54:08,459 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/98c43d8b0d4f48e8ac51c74b03e160e4 2023-03-24 16:54:08,460 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/138590e8ca3f4b2094349e61bcf08942 2023-03-24 16:54:08,461 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/59fcd7e257024bfb99be58415d05388c 2023-03-24 16:54:08,463 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/659ecd47a2af4f909ce886345dfa05ba to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/659ecd47a2af4f909ce886345dfa05ba 2023-03-24 16:54:08,464 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/9db4b1212aac4083887fceffb1614315 2023-03-24 16:54:08,465 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e1da7510d2ea497a8271c66c6608d35a 2023-03-24 16:54:08,467 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7a451b5af9d1494e91e76e4168821595 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7a451b5af9d1494e91e76e4168821595 2023-03-24 16:54:08,468 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/891092a6bcb146aa8aa925a01c27a8ec 2023-03-24 16:54:08,469 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/de9e25c468704651baecd5ece7bbdba7 2023-03-24 16:54:08,470 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d200968bc2784eb092200d9ac97d1a7c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d200968bc2784eb092200d9ac97d1a7c 2023-03-24 16:54:08,472 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8a733253a6cd4f56a8e5d3017e9098dc 2023-03-24 16:54:08,473 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/4c5b3dff952342879e8662554d5dbd5d 2023-03-24 16:54:08,474 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/37663d6795654d81bc8c72ff77e51124 2023-03-24 16:54:08,475 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/e194f41edcb5459bb5d0ceecba63851f 2023-03-24 16:54:08,477 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b479fb87ba57443fbeefce7c58c50560 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b479fb87ba57443fbeefce7c58c50560 2023-03-24 16:54:08,478 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/99d5fa7d11b1482d94b7a2b32dc7b60d 2023-03-24 16:54:08,479 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/64267ede27c4411087de658dca32e7a5 2023-03-24 16:54:08,480 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/0870438766b34268b8264f6d65ca5e47 2023-03-24 16:54:08,482 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7668c6aa3dbb44f1a70f4819b96deb0d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/7668c6aa3dbb44f1a70f4819b96deb0d 2023-03-24 16:54:08,483 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/aa40a82f47b14e48834af2f827ed521b 2023-03-24 16:54:08,484 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1f1b19669bc34cbc9e061d3da5ca91c9 2023-03-24 16:54:08,485 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/ee0abaaca8eb4ff2b4488a95703a43ed 2023-03-24 16:54:08,486 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8eae789c7d634cf4ad5e3609b5ea82cd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8eae789c7d634cf4ad5e3609b5ea82cd 2023-03-24 16:54:08,488 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/1532556eea4e4505ac9a24e129ff6e8e 2023-03-24 16:54:08,489 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d7cee85e454c4346b392e7b4cf63e8bf 2023-03-24 16:54:08,490 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b136f083fdca4a64b6f4f013ace03ebd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b136f083fdca4a64b6f4f013ace03ebd 2023-03-24 16:54:08,491 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/f0ba0c86d06141c4bf90d908197b7938 2023-03-24 16:54:08,493 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/b64fac9878b34b11a98e100929ad9326 2023-03-24 16:54:08,494 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/8351d1006bee41eba8e4961beacbf070 2023-03-24 16:54:08,497 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ce5163776954140a168c2cd8dacfb0e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b4faddb34a214b92b32a50063a5cf311, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/bd45dacf166a4aafa88b6153c86dccc7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/2fb732bca1794c8c94d4cf5355f8481b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1b91286e693741d89ada49188a78b3c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/854c153151974d38ab8fdbaba03a52d3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/16abc890d748484a9399330fa348b42f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35] to archive 2023-03-24 16:54:08,498 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:08,500 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/78dc3ea21ffd4fb9b288b633f1e41d14 2023-03-24 16:54:08,501 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d6e1238ce1c94bfc9330a03f7278a2f0 2023-03-24 16:54:08,502 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ce5163776954140a168c2cd8dacfb0e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ce5163776954140a168c2cd8dacfb0e 2023-03-24 16:54:08,503 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b624564ba48b4d3f80f0fc1ae710e87c 2023-03-24 16:54:08,504 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/69043a59f4aa461bab971168d5836428 2023-03-24 16:54:08,505 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/74e8d167e5354366abb00a467256e518 2023-03-24 16:54:08,506 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/35aed1c4b97343eb8ac0cda9f140dba9 2023-03-24 16:54:08,507 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/418356c29ea34475be4c11f54797d79a 2023-03-24 16:54:08,508 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/0a4339eb0bd740858a98ff9629020d27 2023-03-24 16:54:08,509 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b4faddb34a214b92b32a50063a5cf311 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b4faddb34a214b92b32a50063a5cf311 2023-03-24 16:54:08,510 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/422fe6d01f6246c999cea1ed6cdc7553 2023-03-24 16:54:08,512 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/079149308ead4dec8e72c46949d284e9 2023-03-24 16:54:08,513 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/d35317ba8cff431aa17149c747973891 2023-03-24 16:54:08,514 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/bd45dacf166a4aafa88b6153c86dccc7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/bd45dacf166a4aafa88b6153c86dccc7 2023-03-24 16:54:08,515 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/06d3e9abdef44148a3b6e7b72b2d9848 2023-03-24 16:54:08,516 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1841b9b11b874cd19830f84e72dc01a8 2023-03-24 16:54:08,517 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/29a816732c7743f496326c0693f791da 2023-03-24 16:54:08,518 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/6ba88528c3874aaf9d68ca46f94d0600 2023-03-24 16:54:08,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/2fb732bca1794c8c94d4cf5355f8481b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/2fb732bca1794c8c94d4cf5355f8481b 2023-03-24 16:54:08,520 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/686fa4cc90154ec1bb288583f0dc7d5d 2023-03-24 16:54:08,521 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ca5d68b53f044cc99cc7dc5b8c22dc32 2023-03-24 16:54:08,522 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4aa33ffa62524ce18aa21c6808f3ef30 2023-03-24 16:54:08,523 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1b91286e693741d89ada49188a78b3c9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/1b91286e693741d89ada49188a78b3c9 2023-03-24 16:54:08,524 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/8735f484a9654d1894d7f2b87740ea71 2023-03-24 16:54:08,525 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ad164c225aaa4851b3eaf3268cb977a4 2023-03-24 16:54:08,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/84ecbc6d58e04a5db658ecf3f16d7e5c 2023-03-24 16:54:08,527 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/854c153151974d38ab8fdbaba03a52d3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/854c153151974d38ab8fdbaba03a52d3 2023-03-24 16:54:08,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/748ce96b39014e9d84ba90c29aece371 2023-03-24 16:54:08,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/e2c9dfca252a46f588ff7b58a8c6a973 2023-03-24 16:54:08,530 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/16abc890d748484a9399330fa348b42f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/16abc890d748484a9399330fa348b42f 2023-03-24 16:54:08,531 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/039168e3c99048fb822cb51d384863c4 2023-03-24 16:54:08,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/014abac2a7c7447b930e83736b35f73f 2023-03-24 16:54:08,533 DEBUG [StoreCloser-TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/ef6ae4ff41164e76a006354dade25b35 2023-03-24 16:54:08,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/recovered.edits/868.seqid, newMaxSeqId=868, maxSeqId=1 2023-03-24 16:54:08,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24. 2023-03-24 16:54:08,539 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d3c5b00db9700342afae5fb089181b24: 2023-03-24 16:54:08,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:08,541 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d3c5b00db9700342afae5fb089181b24, regionState=CLOSED 2023-03-24 16:54:08,541 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676848541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676848541"}]},"ts":"1679676848541"} 2023-03-24 16:54:08,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-24 16:54:08,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure d3c5b00db9700342afae5fb089181b24, server=jenkins-hbase4.apache.org,33315,1679676819242 in 1.5960 sec 2023-03-24 16:54:08,545 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-24 16:54:08,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3c5b00db9700342afae5fb089181b24, UNASSIGN in 1.6010 sec 2023-03-24 16:54:08,546 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676848546"}]},"ts":"1679676848546"} 2023-03-24 16:54:08,547 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:54:08,549 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:54:08,551 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6200 sec 2023-03-24 16:54:09,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 16:54:09,044 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-24 16:54:09,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:54:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,057 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,058 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 16:54:09,063 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:09,065 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/recovered.edits] 2023-03-24 16:54:09,068 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/047c618927a34f23a83cb09db9e2fdcd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/047c618927a34f23a83cb09db9e2fdcd 2023-03-24 16:54:09,069 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8177d0607c5e447097cd51be8248d1d0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/A/8177d0607c5e447097cd51be8248d1d0 2023-03-24 16:54:09,072 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/c4cc96715d51487989f06f0aa633d1b9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/c4cc96715d51487989f06f0aa633d1b9 2023-03-24 16:54:09,073 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d0e13b29979f47f783ad0f36064b224b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/B/d0e13b29979f47f783ad0f36064b224b 2023-03-24 16:54:09,075 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4e2cb102f13343979cfa7cc763e56534 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/4e2cb102f13343979cfa7cc763e56534 2023-03-24 16:54:09,076 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b64cbc53cef3402bb927d8271950a10c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/C/b64cbc53cef3402bb927d8271950a10c 2023-03-24 16:54:09,079 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/recovered.edits/868.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24/recovered.edits/868.seqid 2023-03-24 16:54:09,080 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/d3c5b00db9700342afae5fb089181b24 2023-03-24 16:54:09,080 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:54:09,082 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,102 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:54:09,144 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:54:09,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:54:09,146 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676849146"}]},"ts":"9223372036854775807"} 2023-03-24 16:54:09,148 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:54:09,148 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d3c5b00db9700342afae5fb089181b24, NAME => 'TestAcidGuarantees,,1679676822790.d3c5b00db9700342afae5fb089181b24.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:54:09,148 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:54:09,148 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676849148"}]},"ts":"9223372036854775807"} 2023-03-24 16:54:09,150 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:54:09,153 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,154 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 103 msec 2023-03-24 16:54:09,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 16:54:09,161 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-24 16:54:09,177 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:33315-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64fc7b2c-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64fc7b2c-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-825671461_15 at /127.0.0.1:56812 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-825671461_15 at /127.0.0.1:56864 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64fc7b2c-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64fc7b2c-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=572 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 357) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5902 (was 6527) 2023-03-24 16:54:09,188 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=170, AvailableMemoryMB=5901 2023-03-24 16:54:09,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:54:09,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:54:09,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:09,192 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:54:09,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-24 16:54:09,193 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:54:09,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:09,195 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:09,196 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 empty. 2023-03-24 16:54:09,196 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:09,196 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:54:09,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:09,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:09,606 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:54:09,608 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e28bf4621cbd5e09c7a6e132bd2ee3a0, NAME => 'TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:54:09,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e28bf4621cbd5e09c7a6e132bd2ee3a0, disabling compactions & flushes 2023-03-24 16:54:10,014 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. after waiting 0 ms 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,014 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:10,017 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:54:10,018 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676850017"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676850017"}]},"ts":"1679676850017"} 2023-03-24 16:54:10,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:54:10,020 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:54:10,020 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676850020"}]},"ts":"1679676850020"} 2023-03-24 16:54:10,021 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:54:10,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, ASSIGN}] 2023-03-24 16:54:10,028 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, ASSIGN 2023-03-24 16:54:10,028 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:54:10,179 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:10,180 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676850179"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676850179"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676850179"}]},"ts":"1679676850179"} 2023-03-24 16:54:10,181 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:10,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:10,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e28bf4621cbd5e09c7a6e132bd2ee3a0, NAME => 'TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:54:10,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:54:10,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,338 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,340 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A 2023-03-24 16:54:10,340 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A 2023-03-24 16:54:10,340 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:10,340 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName A 2023-03-24 16:54:10,341 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:10,341 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,342 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B 2023-03-24 16:54:10,342 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B 2023-03-24 16:54:10,342 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:10,343 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName B 2023-03-24 16:54:10,343 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:10,343 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,345 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C 2023-03-24 16:54:10,345 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C 2023-03-24 16:54:10,345 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:10,345 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName C 2023-03-24 16:54:10,346 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:10,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:54:10,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:10,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:54:10,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e28bf4621cbd5e09c7a6e132bd2ee3a0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62920376, jitterRate=-0.06241333484649658}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:54:10,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:10,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., pid=18, masterSystemTime=1679676850333 2023-03-24 16:54:10,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:10,358 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:10,358 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676850358"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676850358"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676850358"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676850358"}]},"ts":"1679676850358"} 2023-03-24 16:54:10,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-24 16:54:10,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 in 179 msec 2023-03-24 16:54:10,366 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-24 16:54:10,366 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, ASSIGN in 337 msec 2023-03-24 16:54:10,367 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:54:10,367 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676850367"}]},"ts":"1679676850367"} 2023-03-24 16:54:10,371 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:54:10,375 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:54:10,377 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1850 sec 2023-03-24 16:54:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 16:54:11,299 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-24 16:54:11,300 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x0a607873 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:11,305 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31436cf6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:11,307 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:11,311 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47450, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:11,312 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:54:11,315 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36708, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:54:11,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:54:11,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:54:11,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:11,751 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:54:11,752 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:54:11,752 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:54:11,756 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 16:54:11,768 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, REOPEN/MOVE}] 2023-03-24 16:54:11,769 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, REOPEN/MOVE 2023-03-24 16:54:11,770 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:11,770 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676851770"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676851770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676851770"}]},"ts":"1679676851770"} 2023-03-24 16:54:11,772 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:11,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:11,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e28bf4621cbd5e09c7a6e132bd2ee3a0, disabling compactions & flushes 2023-03-24 16:54:11,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:11,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:11,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. after waiting 0 ms 2023-03-24 16:54:11,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:11,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 16:54:11,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:11,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:11,930 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e28bf4621cbd5e09c7a6e132bd2ee3a0 to self. 2023-03-24 16:54:11,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:11,932 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=CLOSED 2023-03-24 16:54:11,932 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676851932"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676851932"}]},"ts":"1679676851932"} 2023-03-24 16:54:11,935 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-24 16:54:11,936 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 in 163 msec 2023-03-24 16:54:11,936 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=true 2023-03-24 16:54:12,087 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,087 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676852087"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676852087"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676852087"}]},"ts":"1679676852087"} 2023-03-24 16:54:12,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:12,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e28bf4621cbd5e09c7a6e132bd2ee3a0, NAME => 'TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:54:12,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:54:12,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,247 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,248 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A 2023-03-24 16:54:12,248 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A 2023-03-24 16:54:12,248 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:12,251 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName A 2023-03-24 16:54:12,253 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:12,253 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,254 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B 2023-03-24 16:54:12,254 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B 2023-03-24 16:54:12,254 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:12,255 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName B 2023-03-24 16:54:12,255 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:12,255 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,256 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C 2023-03-24 16:54:12,256 DEBUG [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C 2023-03-24 16:54:12,256 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:12,257 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e28bf4621cbd5e09c7a6e132bd2ee3a0 columnFamilyName C 2023-03-24 16:54:12,257 INFO [StoreOpener-e28bf4621cbd5e09c7a6e132bd2ee3a0-1] regionserver.HStore(310): Store=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:12,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:54:12,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e28bf4621cbd5e09c7a6e132bd2ee3a0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65056824, jitterRate=-0.030577778816223145}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:54:12,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:12,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., pid=23, masterSystemTime=1679676852240 2023-03-24 16:54:12,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,265 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,265 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676852265"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676852265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676852265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676852265"}]},"ts":"1679676852265"} 2023-03-24 16:54:12,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-24 16:54:12,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 in 178 msec 2023-03-24 16:54:12,269 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-24 16:54:12,270 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, REOPEN/MOVE in 500 msec 2023-03-24 16:54:12,273 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-24 16:54:12,273 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-03-24 16:54:12,274 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 947 msec 2023-03-24 16:54:12,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-24 16:54:12,284 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x399105f2 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,289 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b3934fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,290 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,294 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bbdba17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,294 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,298 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,298 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,304 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,304 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,308 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,308 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,312 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,313 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,318 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,318 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,323 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,323 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:12,327 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:12,333 DEBUG [hconnection-0x1a2695e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:12,340 DEBUG [hconnection-0x6f5223e5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,342 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47464, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,345 DEBUG [hconnection-0x7bfa29fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,346 DEBUG [hconnection-0x30065b55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,348 DEBUG [hconnection-0x5ace57fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,348 DEBUG [hconnection-0x60a493af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,350 DEBUG [hconnection-0x75702318-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,350 DEBUG [hconnection-0x58ab96f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,351 DEBUG [hconnection-0x46b82f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:12,351 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,352 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,355 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47478, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,355 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47484, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,355 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,355 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47512, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,357 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47536, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,357 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:12,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:12,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:12,361 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:12,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:12,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:12,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,367 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:12,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:12,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:12,372 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,372 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:12,373 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:12,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:12,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676852399, max timestamp=1679676852410Num uniques 859; 2023-03-24 16:54:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676852399, max timestamp=1679676852410Num uniques 859; 2023-03-24 16:54:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676852399, max timestamp=1679676852410Num uniques 859; 2023-03-24 16:54:12,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:12,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:12,431 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324249072d6ccd24191b5b422a650b8dbc2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324249072d6ccd24191b5b422a650b8dbc2 2023-03-24 16:54:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676852430, max timestamp=1679676852436Num uniques 1018; 2023-03-24 16:54:12,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676912435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676852430, max timestamp=1679676852436Num uniques 1018; 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676852430, max timestamp=1679676852436Num uniques 1018; 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:54:12,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:54:12,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1679676852448, max timestamp=1679676852462Num uniques 1077; 2023-03-24 16:54:12,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-03-24 16:54:12,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8094, totalHeapSize=18648, min timestamp=1679676852456, max timestamp=1679676852464Num uniques 1077; 2023-03-24 16:54:12,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49314, new segment data size=8094, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-24 16:54:12,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676852456, max timestamp=1679676852462Num uniques 1027; 2023-03-24 16:54:12,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:54:12,471 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/688a51ede2944a07821177154602076e 2023-03-24 16:54:12,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676912491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676912491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676912493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679676852481, max timestamp=1679676852494Num uniques 1286; 2023-03-24 16:54:12,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679676852481, max timestamp=1679676852494Num uniques 1036; 2023-03-24 16:54:12,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-24 16:54:12,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679676852481, max timestamp=1679676852494Num uniques 1036; 2023-03-24 16:54:12,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-24 16:54:12,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=58896, new segment data size=10806, suffix heap size=161712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1286, new segment cells count=236 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676852515, max timestamp=1679676852523Num uniques 1095; 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676852515, max timestamp=1679676852523Num uniques 1095; 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676852515, max timestamp=1679676852523Num uniques 1095; 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:54:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 16:54:12,532 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/498d22dfefab4bbebe1465a234ae5fdd 2023-03-24 16:54:12,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676852555, max timestamp=1679676852561Num uniques 1054; 2023-03-24 16:54:12,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676852558, max timestamp=1679676852561Num uniques 1054; 2023-03-24 16:54:12,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-24 16:54:12,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676852555, max timestamp=1679676852561Num uniques 1054; 2023-03-24 16:54:12,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-24 16:54:12,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:54:12,578 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f8779a5a52fc4477bf4b74c1a07db058 2023-03-24 16:54:12,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/688a51ede2944a07821177154602076e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e 2023-03-24 16:54:12,591 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e, entries=150, sequenceid=12, filesize=25.1 K 2023-03-24 16:54:12,592 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/498d22dfefab4bbebe1465a234ae5fdd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd 2023-03-24 16:54:12,598 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 16:54:12,607 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f8779a5a52fc4477bf4b74c1a07db058 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058 2023-03-24 16:54:12,620 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 16:54:12,621 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=89.45 KB/91600 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 248ms, sequenceid=12, compaction requested=false 2023-03-24 16:54:12,622 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:12,622 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,622 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:12,622 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:12,622 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:12,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=96.16 KB heapSize=266.31 KB 2023-03-24 16:54:12,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:12,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:12,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:12,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:12,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:12,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676912639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676912639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676912640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676912645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:12,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:12,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676912645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:12,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b570e73[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:12,654 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:12,654 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:12,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:12,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:12,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:12,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:12,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:12,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:12,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:12,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:12,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,675 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:12,675 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:12,695 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248ad573ac53194d93aae47423241ea293 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ad573ac53194d93aae47423241ea293 2023-03-24 16:54:12,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676912746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676912746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676912746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676912747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676912751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,775 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:12,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:12,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:12,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:12,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:12,782 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:12,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:12,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:12,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:12,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:12,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:12,786 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:12,786 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,786 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:12,788 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,788 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:12,788 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:12,788 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:12,788 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:12,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599a150a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:12,793 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:12,793 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,795 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:12,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:12,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:12,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:12,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:12,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:12,800 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:12,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:12,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:12,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:12,800 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:12,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:12,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:12,810 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,811 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:12,811 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:12,811 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,811 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:12,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,911 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:12,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:12,916 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:12,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:12,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:12,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:12,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:12,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:12,926 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,926 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:12,926 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:12,926 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:12,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,930 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:12,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66cd098d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:12,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:12,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:12,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,934 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:12,938 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:12,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:12,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:12,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:12,938 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:12,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:12,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:12,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:12,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:12,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:12,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:12,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:12,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:12,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:12,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:12,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:12,945 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:12,945 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:12,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:12,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676912949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676912951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676912951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676912952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:12,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:12,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676912955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,045 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,055 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,060 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,060 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,060 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:13,060 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d38060[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,066 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,067 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,070 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,071 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,081 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,081 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,081 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,112 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/cf4eea1acca64cceaa70f257c4dcd3d4 2023-03-24 16:54:13,181 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,198 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,198 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,203 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a42c5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,207 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,209 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,212 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,212 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,224 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,224 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676913251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676913254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676913256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676913257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676913258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,324 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:13,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1f4fa4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,346 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,353 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,353 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,359 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,360 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:13,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,360 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,360 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,461 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,465 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,478 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,478 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2210aacb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,485 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,485 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,488 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,488 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,488 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:13,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,494 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,494 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/cae327c5d12446669f3a953d4011a248 2023-03-24 16:54:13,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/89728a6f95db434a877d5992abeb0e3e 2023-03-24 16:54:13,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/cf4eea1acca64cceaa70f257c4dcd3d4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4 2023-03-24 16:54:13,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4, entries=350, sequenceid=129, filesize=52.3 K 2023-03-24 16:54:13,561 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/cae327c5d12446669f3a953d4011a248 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248 2023-03-24 16:54:13,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248, entries=150, sequenceid=129, filesize=11.7 K 2023-03-24 16:54:13,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/89728a6f95db434a877d5992abeb0e3e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e 2023-03-24 16:54:13,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e, entries=150, sequenceid=129, filesize=11.7 K 2023-03-24 16:54:13,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~286.19 KB/293056, currentSize=87.22 KB/89310 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 964ms, sequenceid=129, compaction requested=false 2023-03-24 16:54:13,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:13,594 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,599 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,601 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,602 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,605 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:13,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:13,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,653 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c6aff01f2dbb4527838fefbbcf529e25 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c6aff01f2dbb4527838fefbbcf529e25 2023-03-24 16:54:13,665 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5dde15ad3c1e46b2b7a7e440e634454c 2023-03-24 16:54:13,692 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/eec16b83f43c4e739d87c55ab8e83d1b 2023-03-24 16:54:13,725 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/681399d05a18462cbc03d3bfd89881a9 2023-03-24 16:54:13,730 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5dde15ad3c1e46b2b7a7e440e634454c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c 2023-03-24 16:54:13,736 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c, entries=150, sequenceid=145, filesize=25.3 K 2023-03-24 16:54:13,737 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/eec16b83f43c4e739d87c55ab8e83d1b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b 2023-03-24 16:54:13,744 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b, entries=150, sequenceid=145, filesize=11.8 K 2023-03-24 16:54:13,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/681399d05a18462cbc03d3bfd89881a9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9 2023-03-24 16:54:13,751 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9, entries=150, sequenceid=145, filesize=11.8 K 2023-03-24 16:54:13,752 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 147ms, sequenceid=145, compaction requested=true 2023-03-24 16:54:13,752 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:13,752 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,753 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48aaee69[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,762 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,762 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,768 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,768 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,768 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,770 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,774 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,775 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,777 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,783 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:13,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,783 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:13,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:13,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:13,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:13,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:13,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:13,860 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032452dae45638404a07badec1aa704c7376 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032452dae45638404a07badec1aa704c7376 2023-03-24 16:54:13,876 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/4cfa91dc8f1b47568baa8933fbe04c3e 2023-03-24 16:54:13,883 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:13,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:13,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676913878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676853823, max timestamp=1679676853894Num uniques 859; 2023-03-24 16:54:13,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676913881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676853823, max timestamp=1679676853894Num uniques 859; 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676853823, max timestamp=1679676853894Num uniques 859; 2023-03-24 16:54:13,897 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:13,897 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676913879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:13,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:13,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:13,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,902 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:13,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:13,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:13,904 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:13,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:13,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:13,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:13,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:13,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:13,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/045fc32ca6614b1aa787ec9ab3e7c328 2023-03-24 16:54:13,910 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ce8e5f7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:13,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:13,913 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,914 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:13,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676913908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,917 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:13,917 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:13,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676913915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:13,925 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:13,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:13,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:13,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:13,926 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:13,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:13,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:13,935 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:13,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:13,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:13,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:13,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:13,936 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:13,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:13,936 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:13,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:13,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:13,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f6e28101ede64af6aca40448d96544a7 2023-03-24 16:54:13,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/4cfa91dc8f1b47568baa8933fbe04c3e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e 2023-03-24 16:54:13,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e, entries=400, sequenceid=156, filesize=59.4 K 2023-03-24 16:54:13,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/045fc32ca6614b1aa787ec9ab3e7c328 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328 2023-03-24 16:54:13,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328, entries=400, sequenceid=156, filesize=23.4 K 2023-03-24 16:54:13,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f6e28101ede64af6aca40448d96544a7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7 2023-03-24 16:54:13,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7, entries=400, sequenceid=156, filesize=23.4 K 2023-03-24 16:54:13,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 166ms, sequenceid=156, compaction requested=true 2023-03-24 16:54:13,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:13,988 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:13,989 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:13,989 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:13,989 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:13,989 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:13,990 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 166056 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:13,990 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:13,990 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,990 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=162.2 K 2023-03-24 16:54:13,990 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 688a51ede2944a07821177154602076e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679676852363 2023-03-24 16:54:13,990 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:13,991 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:13,991 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:13,991 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=58.4 K 2023-03-24 16:54:13,991 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cf4eea1acca64cceaa70f257c4dcd3d4, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679676852563 2023-03-24 16:54:13,991 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 498d22dfefab4bbebe1465a234ae5fdd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679676852363 2023-03-24 16:54:13,991 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5dde15ad3c1e46b2b7a7e440e634454c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679676852637 2023-03-24 16:54:13,992 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting cae327c5d12446669f3a953d4011a248, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679676852563 2023-03-24 16:54:13,992 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4cfa91dc8f1b47568baa8933fbe04c3e, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853806 2023-03-24 16:54:13,993 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting eec16b83f43c4e739d87c55ab8e83d1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679676852637 2023-03-24 16:54:13,994 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 045fc32ca6614b1aa787ec9ab3e7c328, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853806 2023-03-24 16:54:14,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:14,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:14,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:14,006 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#119 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:14,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676914016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676914015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676914016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676914022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676914023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,036 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:14,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:14,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:14,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:14,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:14,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:14,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:14,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:14,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:14,054 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#120 average throughput is 0.48 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:14,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:14,063 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/8a419171470a4f9ab3f8eb25da119fe6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8a419171470a4f9ab3f8eb25da119fe6 2023-03-24 16:54:14,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:14,069 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 8a419171470a4f9ab3f8eb25da119fe6(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:14,069 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:14,069 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=12, startTime=1679676853989; duration=0sec 2023-03-24 16:54:14,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:14,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:14,071 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:14,072 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:14,072 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,072 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=58.4 K 2023-03-24 16:54:14,072 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f8779a5a52fc4477bf4b74c1a07db058, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679676852363 2023-03-24 16:54:14,073 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 89728a6f95db434a877d5992abeb0e3e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679676852563 2023-03-24 16:54:14,073 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 681399d05a18462cbc03d3bfd89881a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679676852637 2023-03-24 16:54:14,074 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f6e28101ede64af6aca40448d96544a7, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853806 2023-03-24 16:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,082 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:54:14,092 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248b6e9ee4944d4eccad7257350460933d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248b6e9ee4944d4eccad7257350460933d 2023-03-24 16:54:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:14,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:14,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:14,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a5ce1a1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:14,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:14,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,107 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:14,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:14,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:14,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:14,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,110 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#122 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:14,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:14,116 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:14,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:14,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:14,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:14,116 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:14,118 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,118 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,118 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:14,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:14,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,128 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:14,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,128 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:14,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676914123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676914123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676914129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676914223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,229 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:14,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676914229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:14,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:14,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:14,238 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:14,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:14,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:14,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,240 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:14,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:14,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:14,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:14,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:14,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:14,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:14,244 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:14,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:14,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,247 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3d46d3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:14,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:14,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,253 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:14,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:14,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:14,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:14,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:14,259 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:14,259 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:14,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,262 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,263 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:14,266 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:14,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,267 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,267 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:14,267 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:14,267 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,267 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676914334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676914337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676914337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,367 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:14,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:14,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:14,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:14,376 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:14,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:14,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:14,377 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,377 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:14,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:14,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:14,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:14,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227b2ebb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:14,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:14,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,403 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:14,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:14,413 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:14,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:14,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:14,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:14,413 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:14,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:14,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:14,424 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,424 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:14,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,524 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:14,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:14,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:14,531 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:14,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:14,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:14,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,533 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:14,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:14,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:14,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676914530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,537 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/d6e825bc21c74e89b10d4b42af0d1120 2023-03-24 16:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:14,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676914533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b7b05e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:14,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:14,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:14,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:14,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:14,555 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:14,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:14,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:14,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:14,555 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:14,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,558 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,564 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/a4d8d41b3efe4e6299acc9a8d73e26b9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/a4d8d41b3efe4e6299acc9a8d73e26b9 2023-03-24 16:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:14,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:14,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,569 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:14,569 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:14,568 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/f2af4f61314840cbb4bca7803c215fd1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/f2af4f61314840cbb4bca7803c215fd1 2023-03-24 16:54:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,572 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into a4d8d41b3efe4e6299acc9a8d73e26b9(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:14,572 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:14,572 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=12, startTime=1679676853989; duration=0sec 2023-03-24 16:54:14,572 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:14,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/c2f9f0c8dfd44e4ab41266f3c220630b 2023-03-24 16:54:14,598 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into f2af4f61314840cbb4bca7803c215fd1(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:14,598 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:14,598 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=12, startTime=1679676853988; duration=0sec 2023-03-24 16:54:14,599 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:14,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/517e75a9c727454aac5671a8299423f2 2023-03-24 16:54:14,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/d6e825bc21c74e89b10d4b42af0d1120 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120 2023-03-24 16:54:14,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120, entries=200, sequenceid=182, filesize=32.1 K 2023-03-24 16:54:14,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/c2f9f0c8dfd44e4ab41266f3c220630b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b 2023-03-24 16:54:14,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 16:54:14,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/517e75a9c727454aac5671a8299423f2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2 2023-03-24 16:54:14,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676914639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676914642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:14,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676914643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,669 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:14,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:14,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:14,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:14,682 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:14,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 16:54:14,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 681ms, sequenceid=182, compaction requested=false 2023-03-24 16:54:14,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:14,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:14,702 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:14,703 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:14,703 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:14,754 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240356bec49bd64fbd8154ee2101ef6fb3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240356bec49bd64fbd8154ee2101ef6fb3 2023-03-24 16:54:14,773 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5526ac657bf44d2eb56f390887babf13 2023-03-24 16:54:14,798 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/1578f7410a0f4dd5a344ee1d10db5990 2023-03-24 16:54:14,846 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/e57810097bd747308e66f01d0f0f6aa5 2023-03-24 16:54:14,855 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5526ac657bf44d2eb56f390887babf13 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13 2023-03-24 16:54:14,865 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13, entries=150, sequenceid=195, filesize=25.3 K 2023-03-24 16:54:14,866 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/1578f7410a0f4dd5a344ee1d10db5990 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990 2023-03-24 16:54:14,872 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990, entries=150, sequenceid=195, filesize=11.8 K 2023-03-24 16:54:14,874 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/e57810097bd747308e66f01d0f0f6aa5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5 2023-03-24 16:54:14,893 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5, entries=150, sequenceid=195, filesize=11.8 K 2023-03-24 16:54:14,894 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 191ms, sequenceid=195, compaction requested=true 2023-03-24 16:54:14,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:14,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:14,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:14,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:14,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:14,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:14,898 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:14,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@549b0d20[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:14,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:14,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,905 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:14,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:14,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:14,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:14,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:14,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:14,912 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:14,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:14,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:14,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:14,912 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:14,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:14,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:14,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:14,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:14,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:14,920 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:14,920 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:14,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:14,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:14,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:14,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:15,020 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,027 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,030 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,030 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,031 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:15,031 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,034 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7d2be[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,042 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,046 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,046 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,059 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,059 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:15,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:15,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:15,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:15,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:15,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:15,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:15,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676855096, max timestamp=1679676855142Num uniques 859; 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676855096, max timestamp=1679676855142Num uniques 859; 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679676855096, max timestamp=1679676855142Num uniques 859; 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:15,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-24 16:54:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676915155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,159 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676915157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676915157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,163 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,165 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,168 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,168 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,168 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,168 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,171 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78322a8b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,174 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,174 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,175 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676915173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676915174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,179 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,180 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,193 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,193 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676915258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676915263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676915263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,293 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,297 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,299 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,301 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,303 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c934093[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,306 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,307 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,307 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,307 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,312 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,312 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,316 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,320 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,320 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:15,420 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,423 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60edb22c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,434 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,445 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,451 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,452 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,452 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676915462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676915467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676915467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,519 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303243b33768c7104475c8a50a2765578e574 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243b33768c7104475c8a50a2765578e574 2023-03-24 16:54:15,529 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5f0722ae46234aaea2ed579ed9dece11 2023-03-24 16:54:15,552 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,556 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,559 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,561 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,561 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,561 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,563 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,564 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,564 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d82dead[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,578 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,578 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,593 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,593 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,693 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,697 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,700 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,702 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,702 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,702 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,702 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,703 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,703 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348dd1e4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,719 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,719 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,722 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,727 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,728 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,728 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676915765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676915771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676915772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,828 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,833 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,836 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,839 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,839 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,839 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,839 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,841 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ec883e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,844 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,844 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,846 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,846 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,846 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,846 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,855 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,856 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,868 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,869 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,869 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:15,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,870 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,870 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,870 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/6fe9a6f39d6b4bd8aa892de15674586f 2023-03-24 16:54:15,969 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:15,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:15,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:15,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:15,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:15,972 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:15,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:15,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:15,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:15,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:15,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:15,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:15,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:15,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:15,980 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,981 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:15,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:15,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1d2e9d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:15,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:15,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:15,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:15,988 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:15,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:15,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:15,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:15,988 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:15,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:15,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:15,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:15,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:15,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:15,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:15,998 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:15,998 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:15,998 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:15,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:15,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:16,098 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:16,103 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:16,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:16,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:16,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:16,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:16,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:16,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:16,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:16,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:16,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:16,107 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,107 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:16,107 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:16,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:16,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1f665[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:16,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:16,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:16,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:16,126 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:16,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:16,130 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:16,131 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:16,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:16,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,137 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:16,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:16,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,140 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:16,140 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,140 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,140 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:16,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:16,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676916183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:16,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676916183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,241 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:16,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:16,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:16,245 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:16,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:16,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:16,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:16,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:16,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:16,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:16,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:16,257 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,257 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:16,257 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:16,257 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:16,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:16,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7905e4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:16,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:16,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,263 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:16,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:16,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:16,264 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,264 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:16,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:16,270 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:16,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:16,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:16,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:16,270 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:16,272 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,272 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,272 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:16,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:16,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:16,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:16,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676916274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:16,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676916276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:16,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676916276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:16,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:16,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:16,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,282 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:16,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:16,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:16,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:16,283 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:16,283 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:16,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/00a01be8c2584e43857b35810fe79226 2023-03-24 16:54:16,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5f0722ae46234aaea2ed579ed9dece11 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11 2023-03-24 16:54:16,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11, entries=350, sequenceid=206, filesize=52.6 K 2023-03-24 16:54:16,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/6fe9a6f39d6b4bd8aa892de15674586f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f 2023-03-24 16:54:16,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f, entries=150, sequenceid=206, filesize=11.8 K 2023-03-24 16:54:16,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/00a01be8c2584e43857b35810fe79226 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226 2023-03-24 16:54:16,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226, entries=150, sequenceid=206, filesize=11.8 K 2023-03-24 16:54:16,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1291ms, sequenceid=206, compaction requested=true 2023-03-24 16:54:16,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:16,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:16,382 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:16,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:16,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:54:16,383 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:16,383 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:16,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:16,386 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138676 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:16,386 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:16,386 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,386 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/f2af4f61314840cbb4bca7803c215fd1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=135.4 K 2023-03-24 16:54:16,387 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:16,387 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:16,387 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,387 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8a419171470a4f9ab3f8eb25da119fe6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=47.2 K 2023-03-24 16:54:16,388 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f2af4f61314840cbb4bca7803c215fd1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853811 2023-03-24 16:54:16,388 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8a419171470a4f9ab3f8eb25da119fe6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853811 2023-03-24 16:54:16,389 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d6e825bc21c74e89b10d4b42af0d1120, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676853894 2023-03-24 16:54:16,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:16,389 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5526ac657bf44d2eb56f390887babf13, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679676854010 2023-03-24 16:54:16,389 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting c2f9f0c8dfd44e4ab41266f3c220630b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676853894 2023-03-24 16:54:16,390 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5f0722ae46234aaea2ed579ed9dece11, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855045 2023-03-24 16:54:16,390 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1578f7410a0f4dd5a344ee1d10db5990, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679676854010 2023-03-24 16:54:16,392 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 6fe9a6f39d6b4bd8aa892de15674586f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855079 2023-03-24 16:54:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:16,393 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:16,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:16,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:16,400 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,400 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,402 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:16,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:16,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:16,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:16,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:16,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:16,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:16,417 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,417 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:16,417 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-24 16:54:16,417 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:16,417 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:16,420 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:16,420 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:16,420 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:16,420 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:16,425 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#132 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:16,453 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#131 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:16,496 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/b5c8beed71fd43788c98f9b8d47fd330 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b5c8beed71fd43788c98f9b8d47fd330 2023-03-24 16:54:16,504 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into b5c8beed71fd43788c98f9b8d47fd330(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:16,504 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:16,504 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=12, startTime=1679676856382; duration=0sec 2023-03-24 16:54:16,504 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:16,504 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:16,506 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:16,506 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:16,506 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:16,506 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/a4d8d41b3efe4e6299acc9a8d73e26b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=47.2 K 2023-03-24 16:54:16,506 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a4d8d41b3efe4e6299acc9a8d73e26b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679676853811 2023-03-24 16:54:16,507 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 517e75a9c727454aac5671a8299423f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676853894 2023-03-24 16:54:16,507 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e57810097bd747308e66f01d0f0f6aa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679676854010 2023-03-24 16:54:16,507 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 00a01be8c2584e43857b35810fe79226, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855079 2023-03-24 16:54:16,516 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#134 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:16,901 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324280208318f0b49a8aeb34bafc1dcf59d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324280208318f0b49a8aeb34bafc1dcf59d 2023-03-24 16:54:16,920 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/a6842373ff2d48ef9e62ed9dedfeef66 2023-03-24 16:54:16,926 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/671b6a89d0f446c8acabc903de9e78c2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/671b6a89d0f446c8acabc903de9e78c2 2023-03-24 16:54:16,931 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 671b6a89d0f446c8acabc903de9e78c2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:16,931 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:16,931 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=12, startTime=1679676856382; duration=0sec 2023-03-24 16:54:16,931 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:16,934 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/f815d20728c5484d8791b074675a5dbb 2023-03-24 16:54:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:17,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:17,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676917291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676917292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676917292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,326 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/28368c01b1b24e76899b6708d9598eea as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/28368c01b1b24e76899b6708d9598eea 2023-03-24 16:54:17,335 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 28368c01b1b24e76899b6708d9598eea(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:17,335 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:17,335 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=12, startTime=1679676856382; duration=0sec 2023-03-24 16:54:17,335 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:17,347 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/e057dc7d284a4549b6c5b20a0414c26e 2023-03-24 16:54:17,352 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/a6842373ff2d48ef9e62ed9dedfeef66 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66 2023-03-24 16:54:17,356 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66, entries=150, sequenceid=232, filesize=25.3 K 2023-03-24 16:54:17,357 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/f815d20728c5484d8791b074675a5dbb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb 2023-03-24 16:54:17,363 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb, entries=150, sequenceid=232, filesize=11.8 K 2023-03-24 16:54:17,365 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/e057dc7d284a4549b6c5b20a0414c26e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e 2023-03-24 16:54:17,371 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e, entries=150, sequenceid=232, filesize=11.8 K 2023-03-24 16:54:17,371 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=53.67 KB/54960 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 954ms, sequenceid=232, compaction requested=false 2023-03-24 16:54:17,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:17,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:17,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:17,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:17,376 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,376 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,376 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,376 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:17,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f2cb6d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:17,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:17,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,379 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:17,380 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,380 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:17,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:17,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:17,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:17,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:17,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:17,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,387 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:17,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:17,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:17,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,393 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:17,393 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:17,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:54:17,397 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:17,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:17,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:17,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:17,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:17,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:17,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676917462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676857432, max timestamp=1679676857461Num uniques 859; 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676857432, max timestamp=1679676857461Num uniques 859; 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676857432, max timestamp=1679676857461Num uniques 859; 2023-03-24 16:54:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:17,493 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:17,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:17,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676917491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:17,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:17,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:17,497 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:17,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:17,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:17,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676917495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:17,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:17,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:17,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:17,503 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:17,504 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,504 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebd7b28[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:17,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:17,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:17,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:17,515 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:17,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:17,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:17,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:17,516 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:17,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:17,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,522 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:17,522 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:17,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:17,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676917565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676917615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676917615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,622 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:17,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:17,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:17,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:17,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:17,626 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:17,632 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,632 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a4ac44[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:17,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:17,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:17,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:17,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:17,640 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:17,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:17,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:17,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:17,640 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:17,642 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,642 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,642 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,652 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,652 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:17,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,652 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,652 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,653 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:17,653 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,753 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:17,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:17,756 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:17,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:17,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:17,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:17,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:17,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:17,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:17,761 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,762 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,762 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e4c531[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:17,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:17,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,770 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:17,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:17,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:17,771 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,771 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:17,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:17,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:17,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:17,774 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:17,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:17,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:17,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:17,774 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:17,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676917771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:17,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:17,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,780 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:17,781 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:17,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676917821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:17,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676917821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,859 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324fe8564dab16a40d5a4bb4037518f2634 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fe8564dab16a40d5a4bb4037518f2634 2023-03-24 16:54:17,868 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=248, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5c58684f64c141e5a2f314c5188ea711 2023-03-24 16:54:17,881 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:17,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:17,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:17,888 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:17,888 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,888 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:17,888 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:17,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,890 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,890 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:17,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:17,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:17,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:17,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:17,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:17,894 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,897 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d23d783[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:17,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:17,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,900 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:17,901 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,901 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:17,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:17,907 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:17,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:17,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:17,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:17,907 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:17,912 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,912 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,912 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:17,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:17,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:17,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:17,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:17,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:17,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:17,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:17,920 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:17,920 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:17,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,020 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:18,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:18,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:18,023 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:18,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:18,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:18,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,026 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:18,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:18,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:18,036 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:18,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:18,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18729d34[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:18,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:18,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,043 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:18,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:18,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:18,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:18,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:18,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:18,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:18,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:18,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:18,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,057 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:18,057 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:18,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676918078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676918124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676918125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,157 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:18,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:18,160 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:18,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:18,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:18,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:18,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:18,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:18,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:18,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:18,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:18,166 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6844c4f2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:18,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:18,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,169 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:18,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:18,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:18,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:18,174 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:18,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:18,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:18,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:18,174 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:18,178 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,178 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,178 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:18,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:18,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,183 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:18,183 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:18,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676918194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,197 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676918203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,206 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:18,284 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:18,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:18,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:18,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:18,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:18,287 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:18,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:18,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:18,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/8483d8ffe5954b2299a039a9f4eef017 2023-03-24 16:54:18,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:18,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:18,297 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,297 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:18,297 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:18,297 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,299 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d6c4986[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:18,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:18,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,303 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:18,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:18,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:18,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:18,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/60c5ac01ba4b46528c857c9a4285ad78 2023-03-24 16:54:18,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:18,308 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:18,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:18,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:18,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:18,308 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:18,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/5c58684f64c141e5a2f314c5188ea711 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711 2023-03-24 16:54:18,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:18,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,313 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711, entries=250, sequenceid=248, filesize=38.9 K 2023-03-24 16:54:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:18,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/8483d8ffe5954b2299a039a9f4eef017 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017 2023-03-24 16:54:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:18,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,320 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:18,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:18,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:18,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017, entries=150, sequenceid=248, filesize=11.8 K 2023-03-24 16:54:18,320 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:18,320 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:18,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:18,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/60c5ac01ba4b46528c857c9a4285ad78 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78 2023-03-24 16:54:18,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:18,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78, entries=150, sequenceid=248, filesize=11.8 K 2023-03-24 16:54:18,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 929ms, sequenceid=248, compaction requested=true 2023-03-24 16:54:18,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:18,326 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:18,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:18,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:18,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:18,327 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:18,328 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:18,328 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:18,328 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,328 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/28368c01b1b24e76899b6708d9598eea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=89.8 K 2023-03-24 16:54:18,329 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:18,329 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 28368c01b1b24e76899b6708d9598eea, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855079 2023-03-24 16:54:18,329 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:18,329 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,329 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b5c8beed71fd43788c98f9b8d47fd330, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=35.5 K 2023-03-24 16:54:18,329 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a6842373ff2d48ef9e62ed9dedfeef66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679676855153 2023-03-24 16:54:18,329 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b5c8beed71fd43788c98f9b8d47fd330, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855079 2023-03-24 16:54:18,330 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5c58684f64c141e5a2f314c5188ea711, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:18,330 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f815d20728c5484d8791b074675a5dbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679676855153 2023-03-24 16:54:18,330 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8483d8ffe5954b2299a039a9f4eef017, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:18,341 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:18,342 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#140 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:18,392 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/78dc56b886674b76b2e026f24bea8419 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/78dc56b886674b76b2e026f24bea8419 2023-03-24 16:54:18,398 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 78dc56b886674b76b2e026f24bea8419(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:18,398 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:18,398 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=13, startTime=1679676858326; duration=0sec 2023-03-24 16:54:18,399 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:18,399 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:18,400 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:18,400 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:18,400 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,400 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/671b6a89d0f446c8acabc903de9e78c2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=35.5 K 2023-03-24 16:54:18,400 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 671b6a89d0f446c8acabc903de9e78c2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679676855079 2023-03-24 16:54:18,401 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e057dc7d284a4549b6c5b20a0414c26e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679676855153 2023-03-24 16:54:18,401 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 60c5ac01ba4b46528c857c9a4285ad78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:18,407 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#142 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:18,418 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/dd9cf6d35979457bb282149250596e5e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/dd9cf6d35979457bb282149250596e5e 2023-03-24 16:54:18,420 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:18,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:18,423 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into dd9cf6d35979457bb282149250596e5e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:18,423 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:18,423 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=13, startTime=1679676858327; duration=0sec 2023-03-24 16:54:18,423 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:18,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:18,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:18,424 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:18,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:18,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:18,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:18,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:18,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:18,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:18,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:18,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:18,428 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=114.05 KB heapSize=320.53 KB 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:18,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:18,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:18,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676918610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676918628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676918629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676918713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:18,772 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/efe30d95bab4480bb7befab946df0a9b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/efe30d95bab4480bb7befab946df0a9b 2023-03-24 16:54:18,777 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into efe30d95bab4480bb7befab946df0a9b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:18,777 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:18,777 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=13, startTime=1679676858327; duration=0sec 2023-03-24 16:54:18,777 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:18,841 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032455310e7b42034a19a8d9ea3ad2be7d5f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455310e7b42034a19a8d9ea3ad2be7d5f 2023-03-24 16:54:18,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676918916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:19,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47542 deadline: 1679676919219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,250 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/c52e405c45ad4aa688164f9b3025e830 2023-03-24 16:54:19,273 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/10d4c66008134c6eb08318033bcfbe5b 2023-03-24 16:54:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676919635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676919636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,688 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/15906f456c5f4f8e94a3e54eddddd2e2 2023-03-24 16:54:19,695 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/c52e405c45ad4aa688164f9b3025e830 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830 2023-03-24 16:54:19,707 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830, entries=150, sequenceid=283, filesize=25.4 K 2023-03-24 16:54:19,708 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/10d4c66008134c6eb08318033bcfbe5b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b 2023-03-24 16:54:19,713 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b, entries=150, sequenceid=283, filesize=11.9 K 2023-03-24 16:54:19,721 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/15906f456c5f4f8e94a3e54eddddd2e2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2 2023-03-24 16:54:19,726 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2, entries=150, sequenceid=283, filesize=11.9 K 2023-03-24 16:54:19,727 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~320.48 KB/328176, currentSize=67.09 KB/68700 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1299ms, sequenceid=283, compaction requested=false 2023-03-24 16:54:19,727 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:19,727 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:19,731 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,732 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,732 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e400867[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:19,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:19,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:19,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:19,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:19,737 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:19,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:19,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:19,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:19,741 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:19,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:19,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:19,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:19,741 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:19,743 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,744 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:19,744 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:19,745 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:19,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,750 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,750 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,750 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:19,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:19,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:19,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:19,751 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:19,751 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:19,782 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242c3b1ad3864f4ad1a26a24bb858dc7d9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c3b1ad3864f4ad1a26a24bb858dc7d9 2023-03-24 16:54:19,793 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/6c1d30016e6c4cb88f7b06b350673f05 2023-03-24 16:54:19,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676919804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/e5a2efea41a44519ac60e209a09b1864 2023-03-24 16:54:19,851 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:19,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/bf6444da0c8b499b973fb1343dd7feb2 2023-03-24 16:54:19,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:19,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:19,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/6c1d30016e6c4cb88f7b06b350673f05 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05 2023-03-24 16:54:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:19,860 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:19,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:19,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:19,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:19,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05, entries=150, sequenceid=298, filesize=25.4 K 2023-03-24 16:54:19,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/e5a2efea41a44519ac60e209a09b1864 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864 2023-03-24 16:54:19,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864, entries=150, sequenceid=298, filesize=11.9 K 2023-03-24 16:54:19,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/bf6444da0c8b499b973fb1343dd7feb2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2 2023-03-24 16:54:19,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,870 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,870 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:19,871 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:19,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:19,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:19,874 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:19,874 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:19,874 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:19,875 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,875 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:19,875 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:19,875 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:19,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2, entries=150, sequenceid=298, filesize=11.9 K 2023-03-24 16:54:19,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,879 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 145ms, sequenceid=298, compaction requested=true 2023-03-24 16:54:19,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:19,879 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:19,879 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:54:19,879 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 16:54:19,879 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:19,880 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:19,883 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:19,883 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:19,883 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,883 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/dd9cf6d35979457bb282149250596e5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=35.9 K 2023-03-24 16:54:19,883 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78364 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:19,884 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:19,884 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,884 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting dd9cf6d35979457bb282149250596e5e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:19,884 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,884 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/78dc56b886674b76b2e026f24bea8419, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=76.5 K 2023-03-24 16:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,884 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 15906f456c5f4f8e94a3e54eddddd2e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679676857477 2023-03-24 16:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,885 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting bf6444da0c8b499b973fb1343dd7feb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,885 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 78dc56b886674b76b2e026f24bea8419, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,886 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c52e405c45ad4aa688164f9b3025e830, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679676857477 2023-03-24 16:54:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,887 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6c1d30016e6c4cb88f7b06b350673f05, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:19,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:19,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d7a9a6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:19,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:19,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,888 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:19,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:19,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:19,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:19,890 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,890 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:19,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:19,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:19,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:19,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:19,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:19,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:19,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:19,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:19,900 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:19,902 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:19,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:19,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:19,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:19,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:19,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:19,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:19,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:19,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:19,909 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:19,909 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#149 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:19,909 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:19,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:19,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:19,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:19,944 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/9ac548d864f14b01a63cea95f628f8ff as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/9ac548d864f14b01a63cea95f628f8ff 2023-03-24 16:54:19,952 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 9ac548d864f14b01a63cea95f628f8ff(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:19,952 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:19,952 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=13, startTime=1679676859879; duration=0sec 2023-03-24 16:54:19,952 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:19,952 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:19,953 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:19,954 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:19,954 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:19,954 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/efe30d95bab4480bb7befab946df0a9b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=35.9 K 2023-03-24 16:54:19,954 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting efe30d95bab4480bb7befab946df0a9b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679676857287 2023-03-24 16:54:19,960 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 10d4c66008134c6eb08318033bcfbe5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679676857477 2023-03-24 16:54:19,961 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e5a2efea41a44519ac60e209a09b1864, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:19,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:19,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676919961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:19,981 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324716116492f594aa08b50291b60e3827d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324716116492f594aa08b50291b60e3827d 2023-03-24 16:54:19,981 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#152 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:20,001 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/1b491398dfbb4f9bbd1afc79efc88389 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/1b491398dfbb4f9bbd1afc79efc88389 2023-03-24 16:54:20,008 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 1b491398dfbb4f9bbd1afc79efc88389(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:20,008 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:20,008 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=13, startTime=1679676859879; duration=0sec 2023-03-24 16:54:20,009 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:20,009 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,016 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,017 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,017 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,019 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,020 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/610257b9c15346f78bd7701e53572513 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/610257b9c15346f78bd7701e53572513 2023-03-24 16:54:20,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,023 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,023 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,023 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,023 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,023 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,024 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,027 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,027 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfa985a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,030 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,034 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 610257b9c15346f78bd7701e53572513(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:20,034 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:20,034 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=13, startTime=1679676859879; duration=0sec 2023-03-24 16:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,034 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,035 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,035 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,038 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,038 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,038 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,045 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,045 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:20,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676920072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,145 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,149 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,150 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,150 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,151 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,152 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1335ce3a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,158 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,165 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,166 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,172 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,172 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:20,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,272 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,275 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676920274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,277 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,279 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,279 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,279 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,279 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d08ef0b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,286 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,289 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,289 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,298 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,298 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,398 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,401 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,403 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,410 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@221df995[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,413 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,419 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/7b396544a281438ca72c56ca6683a855 2023-03-24 16:54:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,419 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,419 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,422 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,422 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,430 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,430 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,530 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,533 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,538 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,543 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,543 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,543 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,543 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a8d8dc4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,554 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,554 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,556 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,556 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,556 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,558 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,561 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,561 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676920578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,662 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,665 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,669 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,669 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,669 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,669 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f4c6b1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,678 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,678 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,684 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,684 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:20,784 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,787 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,789 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,792 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,792 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,792 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:20,792 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:20,799 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:20,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6e17f3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:20,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:20,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:20,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:20,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:20,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:20,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:20,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:20,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:20,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:20,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:20,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:20,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:20,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:20,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:20,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,815 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:20,815 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:20,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:20,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/115a287367694dd19f92d9aff7432718 2023-03-24 16:54:20,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/59c4fecc7f44471c940d6db59d970a64 2023-03-24 16:54:20,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/7b396544a281438ca72c56ca6683a855 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855 2023-03-24 16:54:20,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855, entries=200, sequenceid=317, filesize=32.3 K 2023-03-24 16:54:20,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/115a287367694dd19f92d9aff7432718 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718 2023-03-24 16:54:20,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718, entries=150, sequenceid=317, filesize=11.9 K 2023-03-24 16:54:20,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/59c4fecc7f44471c940d6db59d970a64 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64 2023-03-24 16:54:20,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64, entries=150, sequenceid=317, filesize=11.9 K 2023-03-24 16:54:20,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 977ms, sequenceid=317, compaction requested=false 2023-03-24 16:54:20,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:20,915 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:20,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:20,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:20,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:20,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:20,918 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:20,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:20,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:20,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:20,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:20,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:20,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:20,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:20,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:20,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:20,945 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:20,945 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:20,945 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:20,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:20,978 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032426ce549d61f54988a7382a152d74c22a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032426ce549d61f54988a7382a152d74c22a 2023-03-24 16:54:20,986 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=333, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/2d2eefe9ad2647039c8efbbb132250e7 2023-03-24 16:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:21,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:21,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676921124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676921227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,407 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/b2a4805dcc194a6b824205e3beca1a38 2023-03-24 16:54:21,430 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/7b7ed8f58f8a46b49bb4fd54bef6374f 2023-03-24 16:54:21,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676921431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,436 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/2d2eefe9ad2647039c8efbbb132250e7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7 2023-03-24 16:54:21,443 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7, entries=150, sequenceid=333, filesize=25.4 K 2023-03-24 16:54:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/b2a4805dcc194a6b824205e3beca1a38 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38 2023-03-24 16:54:21,450 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38, entries=150, sequenceid=333, filesize=11.9 K 2023-03-24 16:54:21,451 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/7b7ed8f58f8a46b49bb4fd54bef6374f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f 2023-03-24 16:54:21,457 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f, entries=150, sequenceid=333, filesize=11.9 K 2023-03-24 16:54:21,472 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 527ms, sequenceid=333, compaction requested=true 2023-03-24 16:54:21,472 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:21,472 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:21,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10729c9a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:21,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:21,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,478 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:21,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,479 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:21,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:21,484 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:21,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:21,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:21,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:21,484 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:21,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,488 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:21,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:21,494 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:21,494 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:21,493 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,594 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:21,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:21,601 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:21,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:21,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:21,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:21,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,601 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:21,603 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:21,603 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,603 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:21,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:21,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:21,606 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:21,606 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:21,606 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:21,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,638 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c9822ffd91cc4dc1a95f6c07fab7ee3c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c9822ffd91cc4dc1a95f6c07fab7ee3c 2023-03-24 16:54:21,649 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:21,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676921662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676921663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,667 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/eec0bf3313fa4d1e88710d2cfad10ae0 2023-03-24 16:54:21,702 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d 2023-03-24 16:54:21,719 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/fab81fa1df04470ab59a80466374b62b 2023-03-24 16:54:21,725 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/eec0bf3313fa4d1e88710d2cfad10ae0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0 2023-03-24 16:54:21,730 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0, entries=150, sequenceid=352, filesize=25.4 K 2023-03-24 16:54:21,732 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d 2023-03-24 16:54:21,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676921734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,743 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d, entries=150, sequenceid=352, filesize=11.9 K 2023-03-24 16:54:21,744 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/fab81fa1df04470ab59a80466374b62b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b 2023-03-24 16:54:21,750 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b, entries=150, sequenceid=352, filesize=11.9 K 2023-03-24 16:54:21,751 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 145ms, sequenceid=352, compaction requested=true 2023-03-24 16:54:21,751 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:21,751 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:21,755 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7677c0fd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:21,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:21,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:21,759 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,759 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:21,760 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:21,761 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,761 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:21,766 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:21,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:21,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:21,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:21,766 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:21,768 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,768 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,768 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,769 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,770 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:21,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:21,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:21,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:21,775 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:21,775 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:21,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:21,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:21,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,806 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a04e8bb90f1f4c1085a99151312759d7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a04e8bb90f1f4c1085a99151312759d7 2023-03-24 16:54:21,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676921804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676921806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,876 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:21,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:21,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:21,880 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:21,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:21,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:21,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:21,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,882 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:21,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:21,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:21,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:21,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:21,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:21,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:21,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,887 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,887 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,887 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a741cc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:21,888 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:21,888 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,893 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:21,895 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:21,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:21,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:21,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:21,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:21,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:21,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:21,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:21,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:21,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,908 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,911 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:21,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:21,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:21,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:21,912 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:21,912 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:21,911 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:21,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:21,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676921910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:21,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676921910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,012 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:22,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:22,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:22,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:22,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:22,020 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:22,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:22,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:22,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,022 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:22,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:22,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:22,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:22,029 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,029 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:22,029 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:22,030 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:22,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:22,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5b1344[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:22,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:22,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:22,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:22,043 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:22,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:22,046 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:22,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:22,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:22,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:22,047 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:22,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:22,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:22,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:22,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:22,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:22,056 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:22,056 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47536 deadline: 1679676922114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676922114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,157 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:22,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:22,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:22,160 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:22,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:22,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:22,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,163 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:22,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:22,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:22,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:22,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec9c0d9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:22,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:22,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:22,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:22,181 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:22,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:22,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:22,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:22,181 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:22,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:22,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:22,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:22,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:22,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:22,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:22,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:22,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:22,192 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:22,192 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:22,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:22,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676922202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,206 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:22,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676922213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,216 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:22,219 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/9382032001ee4f738a3a80a4e3244b8e 2023-03-24 16:54:22,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/0e5f0dc7c3394b5b81b393282f5c110e 2023-03-24 16:54:22,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676922241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/af3ab0e88d31482f91db322bf93d1846 2023-03-24 16:54:22,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/9382032001ee4f738a3a80a4e3244b8e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e 2023-03-24 16:54:22,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e, entries=150, sequenceid=366, filesize=25.4 K 2023-03-24 16:54:22,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/0e5f0dc7c3394b5b81b393282f5c110e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e 2023-03-24 16:54:22,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e, entries=150, sequenceid=366, filesize=11.9 K 2023-03-24 16:54:22,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/af3ab0e88d31482f91db322bf93d1846 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846 2023-03-24 16:54:22,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846, entries=150, sequenceid=366, filesize=11.9 K 2023-03-24 16:54:22,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 493ms, sequenceid=366, compaction requested=true 2023-03-24 16:54:22,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:22,266 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:22,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:22,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:22,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:22,266 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:22,269 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:22,269 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:22,269 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,269 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:22,269 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/1b491398dfbb4f9bbd1afc79efc88389, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=134.5 K 2023-03-24 16:54:22,269 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:22,270 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,270 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/610257b9c15346f78bd7701e53572513, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=60.0 K 2023-03-24 16:54:22,270 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1b491398dfbb4f9bbd1afc79efc88389, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:22,270 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 610257b9c15346f78bd7701e53572513, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:22,270 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7b396544a281438ca72c56ca6683a855, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679676859784 2023-03-24 16:54:22,271 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 115a287367694dd19f92d9aff7432718, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679676859791 2023-03-24 16:54:22,272 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 2d2eefe9ad2647039c8efbbb132250e7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679676859933 2023-03-24 16:54:22,272 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b2a4805dcc194a6b824205e3beca1a38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679676859933 2023-03-24 16:54:22,272 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting eec0bf3313fa4d1e88710d2cfad10ae0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676861111 2023-03-24 16:54:22,272 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 186ddcceaf2b4f1cbe3b3cf3b51fa85d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676861111 2023-03-24 16:54:22,272 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9382032001ee4f738a3a80a4e3244b8e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:22,273 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 0e5f0dc7c3394b5b81b393282f5c110e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:22,287 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#165 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:22,290 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#164 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:22,292 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:22,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:22,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:22,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:22,300 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:22,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:22,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:22,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:22,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:22,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:22,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:22,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:22,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:22,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:22,307 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:22,307 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:22,311 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/a941c2c1e27c4d28aa05ce1d0a4f5d86 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/a941c2c1e27c4d28aa05ce1d0a4f5d86 2023-03-24 16:54:22,321 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into a941c2c1e27c4d28aa05ce1d0a4f5d86(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:22,321 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:22,321 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=11, startTime=1679676862266; duration=0sec 2023-03-24 16:54:22,321 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:22,321 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:22,324 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:22,324 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:22,324 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:22,324 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/9ac548d864f14b01a63cea95f628f8ff, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=60.0 K 2023-03-24 16:54:22,325 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9ac548d864f14b01a63cea95f628f8ff, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679676858598 2023-03-24 16:54:22,326 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 59c4fecc7f44471c940d6db59d970a64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679676859791 2023-03-24 16:54:22,326 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7b7ed8f58f8a46b49bb4fd54bef6374f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679676859933 2023-03-24 16:54:22,329 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fab81fa1df04470ab59a80466374b62b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676861111 2023-03-24 16:54:22,330 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting af3ab0e88d31482f91db322bf93d1846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:22,350 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#167 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:22,353 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032482a8812bbdbe4c13bd3b2fed72ade51d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482a8812bbdbe4c13bd3b2fed72ade51d 2023-03-24 16:54:22,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:22,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:22,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676922434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676922437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47484 deadline: 1679676922541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676922544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,749 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/2ffac5d61df14c3fbaf61073ecea6552 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2ffac5d61df14c3fbaf61073ecea6552 2023-03-24 16:54:22,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676922744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:22,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676922748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:22,762 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 2ffac5d61df14c3fbaf61073ecea6552(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:22,762 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:22,762 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=11, startTime=1679676862266; duration=0sec 2023-03-24 16:54:22,762 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:22,785 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/25401791689c47a1b3c26cf79347f7e8 2023-03-24 16:54:22,818 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/042587c995694e378c4dda88570dd1ef as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/042587c995694e378c4dda88570dd1ef 2023-03-24 16:54:22,830 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 042587c995694e378c4dda88570dd1ef(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:22,830 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:22,830 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=11, startTime=1679676862266; duration=0sec 2023-03-24 16:54:22,830 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:23,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676923052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676923053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,235 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/fa94f90683eb40e1b28ca25bc925fbe2 2023-03-24 16:54:23,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:23,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676923247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676923555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:23,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676923558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,674 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/95d0ecc220c34bcf8bf29abe33dfabd1 2023-03-24 16:54:23,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/25401791689c47a1b3c26cf79347f7e8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8 2023-03-24 16:54:23,683 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8, entries=150, sequenceid=384, filesize=25.4 K 2023-03-24 16:54:23,686 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/fa94f90683eb40e1b28ca25bc925fbe2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2 2023-03-24 16:54:23,690 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2, entries=150, sequenceid=384, filesize=11.9 K 2023-03-24 16:54:23,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/95d0ecc220c34bcf8bf29abe33dfabd1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1 2023-03-24 16:54:23,695 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1, entries=150, sequenceid=384, filesize=11.9 K 2023-03-24 16:54:23,696 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1390ms, sequenceid=384, compaction requested=false 2023-03-24 16:54:23,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:23,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:23,698 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67191d5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:23,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:23,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:23,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:23,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:23,710 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:23,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:23,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:23,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:23,711 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:23,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:23,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:23,715 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:23,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:23,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:23,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:23,720 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:23,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:23,720 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:23,720 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:23,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:23,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:23,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:23,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,820 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:23,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:23,824 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:23,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:23,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:23,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:23,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:23,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:23,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:23,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:23,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:23,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:23,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:23,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:23,828 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:23,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:23,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:23,855 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032464a19edce4be483cb987c26d6744b2e1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032464a19edce4be483cb987c26d6744b2e1 2023-03-24 16:54:24,271 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/9ab225344a4c4309bf5b0750710ff22a 2023-03-24 16:54:24,310 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/e1393c4c13254e0eaa780bef5c347e2b 2023-03-24 16:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:24,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:24,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676924580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676924581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676924683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676924684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,725 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/3490b5e91ab44247b1b16eef5ac2621e 2023-03-24 16:54:24,730 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/9ab225344a4c4309bf5b0750710ff22a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a 2023-03-24 16:54:24,734 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a, entries=150, sequenceid=401, filesize=25.4 K 2023-03-24 16:54:24,735 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/e1393c4c13254e0eaa780bef5c347e2b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b 2023-03-24 16:54:24,751 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b, entries=150, sequenceid=401, filesize=11.9 K 2023-03-24 16:54:24,752 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/3490b5e91ab44247b1b16eef5ac2621e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e 2023-03-24 16:54:24,768 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e, entries=150, sequenceid=401, filesize=11.9 K 2023-03-24 16:54:24,769 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 941ms, sequenceid=401, compaction requested=true 2023-03-24 16:54:24,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:24,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:24,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,772 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8509f6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:24,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:24,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:24,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:24,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:24,780 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:24,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:24,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:24,780 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:24,780 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:24,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,782 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:24,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,786 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,786 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:24,786 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:24,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:24,786 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:24,786 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:24,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:24,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:24,786 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:24,787 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:24,787 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:24,887 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:24,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:24,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:54:24,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:24,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:24,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:24,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:24,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:24,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:24,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:24,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:24,898 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:24,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:24,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:24,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:24,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,902 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:24,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:24,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:24,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:24,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:24,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676924906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:24,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:24,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676924909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,910 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d6e0f1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:24,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:24,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:24,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:24,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:24,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:24,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:24,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:24,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:24,918 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:24,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:24,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:24,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:24,918 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:24,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:24,929 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:24,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:24,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:24,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:24,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:24,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:24,939 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:24,939 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:24,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676925009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676925011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,039 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,042 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,046 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,048 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49192e1a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,050 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,061 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,061 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,075 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,075 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,175 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,179 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,179 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,183 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,186 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ece1d5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,186 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,199 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,199 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,205 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,205 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676925215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676925216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676925263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,266 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:25,305 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,314 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240f999d4aba08409ea678c8fa33bb703f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240f999d4aba08409ea678c8fa33bb703f 2023-03-24 16:54:25,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,315 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,316 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,320 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,324 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,324 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,324 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,325 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,327 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30d59ae7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,332 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,339 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,339 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,347 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,347 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,447 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,450 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,450 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,455 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,455 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,455 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,455 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,457 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efa3564[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,461 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,465 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,465 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,475 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,475 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676925518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:25,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676925519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,575 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,578 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,580 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e13ab5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,598 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,598 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,598 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,698 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,702 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,703 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,705 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,705 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,705 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,705 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad51901[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,714 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,714 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,720 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,720 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,722 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=421, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/799029eda5f2410ea47365cd9b91d27f 2023-03-24 16:54:25,820 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,823 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,823 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,827 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,827 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,827 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,827 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,830 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cba41e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,836 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,836 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,843 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,843 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,843 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,944 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:25,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:25,947 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:25,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:25,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:25,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,948 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:25,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:25,953 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,953 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:25,953 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:25,953 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,955 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6905feb6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:25,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:25,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:25,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,960 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:25,962 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:25,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:25,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:25,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:25,962 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:25,963 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:25,963 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,964 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,964 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:25,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:25,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:25,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:25,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:25,968 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:25,968 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:25,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676926022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676926024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,068 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:26,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:26,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:26,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:26,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:26,073 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,076 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:26,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:26,077 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,077 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:26,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:26,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:26,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1fe99a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:26,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:26,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:26,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:26,083 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,083 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,084 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:26,087 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:26,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:26,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:26,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:26,087 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:26,089 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,089 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,089 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,090 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:26,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:26,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,093 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:26,093 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:26,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/7c38850b78434498bdbdb5534b0b6bdd 2023-03-24 16:54:26,193 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:26,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:26,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:26,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:26,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:26,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:26,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:26,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:26,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:26,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:26,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:26,211 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:26,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7840e2c9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:26,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:26,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:26,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:26,220 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:26,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:26,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:26,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:26,220 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:26,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,225 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:26,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:26,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,228 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:26,228 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:26,329 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:26,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:26,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:26,332 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:26,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:26,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:26,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:26,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:26,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:26,336 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,336 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:26,336 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:26,336 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:26,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:26,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:26,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5300528b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:26,340 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:26,340 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,344 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:26,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:26,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:26,350 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:26,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:26,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:26,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:26,350 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:26,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:26,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:26,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,357 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:26,357 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:26,457 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:26,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:26,460 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:26,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:26,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:26,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:26,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:26,464 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,465 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:26,465 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:26,465 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:26,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:26,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:26,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:26,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,467 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,467 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,467 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b6f613[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:26,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:26,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,470 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:26,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,472 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:26,474 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:26,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:26,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:26,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:26,474 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:26,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,476 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:26,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,476 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:26,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,478 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:26,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:26,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:26,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:26,487 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:26,487 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:26,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:26,487 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:26,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/2305f15fa61f4f0bb66f6386bb2a37b7 2023-03-24 16:54:26,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/799029eda5f2410ea47365cd9b91d27f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f 2023-03-24 16:54:26,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f, entries=200, sequenceid=421, filesize=32.3 K 2023-03-24 16:54:26,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/7c38850b78434498bdbdb5534b0b6bdd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd 2023-03-24 16:54:26,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd, entries=150, sequenceid=421, filesize=11.9 K 2023-03-24 16:54:26,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/2305f15fa61f4f0bb66f6386bb2a37b7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7 2023-03-24 16:54:26,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7, entries=150, sequenceid=421, filesize=11.9 K 2023-03-24 16:54:26,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1680ms, sequenceid=421, compaction requested=true 2023-03-24 16:54:26,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:26,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:26,567 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:26,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:26,567 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:26,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:26,569 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:26,569 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:26,569 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,569 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2ffac5d61df14c3fbaf61073ecea6552, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=109.3 K 2023-03-24 16:54:26,569 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:26,569 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:26,569 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,570 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/a941c2c1e27c4d28aa05ce1d0a4f5d86, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=48.2 K 2023-03-24 16:54:26,570 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a941c2c1e27c4d28aa05ce1d0a4f5d86, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fa94f90683eb40e1b28ca25bc925fbe2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679676861795 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 2ffac5d61df14c3fbaf61073ecea6552, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e1393c4c13254e0eaa780bef5c347e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679676862429 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 25401791689c47a1b3c26cf79347f7e8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679676861795 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7c38850b78434498bdbdb5534b0b6bdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864580 2023-03-24 16:54:26,571 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9ab225344a4c4309bf5b0750710ff22a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679676862429 2023-03-24 16:54:26,572 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 799029eda5f2410ea47365cd9b91d27f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864578 2023-03-24 16:54:26,579 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:26,587 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:26,590 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#177 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:26,590 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:26,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:26,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:26,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,593 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:26,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:26,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:26,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:26,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:26,599 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/21281d9ad18a4f0e941daebe88ba8903 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/21281d9ad18a4f0e941daebe88ba8903 2023-03-24 16:54:26,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:26,603 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:26,604 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:54:26,604 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 21281d9ad18a4f0e941daebe88ba8903(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:26,605 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:26,605 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=12, startTime=1679676866567; duration=0sec 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:26,605 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:26,605 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:26,605 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:26,606 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:26,606 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:26,606 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:26,607 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/042587c995694e378c4dda88570dd1ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=48.2 K 2023-03-24 16:54:26,607 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 042587c995694e378c4dda88570dd1ef, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679676861659 2023-03-24 16:54:26,607 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 95d0ecc220c34bcf8bf29abe33dfabd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679676861795 2023-03-24 16:54:26,608 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3490b5e91ab44247b1b16eef5ac2621e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679676862429 2023-03-24 16:54:26,609 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2305f15fa61f4f0bb66f6386bb2a37b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864580 2023-03-24 16:54:26,620 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#178 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:26,633 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303247a512cad786d4372b009643ddb605230 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247a512cad786d4372b009643ddb605230 2023-03-24 16:54:26,636 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f1d2948f8cb3442cacd5986a93f66bae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f1d2948f8cb3442cacd5986a93f66bae 2023-03-24 16:54:26,644 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into f1d2948f8cb3442cacd5986a93f66bae(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:26,644 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:26,644 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=12, startTime=1679676866567; duration=0sec 2023-03-24 16:54:26,645 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:27,022 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/4241d937eb5b4f4fab95f5093fddbfc2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4241d937eb5b4f4fab95f5093fddbfc2 2023-03-24 16:54:27,027 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 4241d937eb5b4f4fab95f5093fddbfc2(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:27,027 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:27,027 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=12, startTime=1679676866567; duration=0sec 2023-03-24 16:54:27,027 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:27,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:27,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:27,041 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/6d0d3c092c65484b95019f168b38617d 2023-03-24 16:54:27,066 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/b9457d23598f491ba9bf89f158fb17de 2023-03-24 16:54:27,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,477 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/d3124a90a6d04bc0b4d51640a343b504 2023-03-24 16:54:27,482 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/6d0d3c092c65484b95019f168b38617d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d 2023-03-24 16:54:27,487 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d, entries=150, sequenceid=434, filesize=25.4 K 2023-03-24 16:54:27,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/b9457d23598f491ba9bf89f158fb17de as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de 2023-03-24 16:54:27,493 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de, entries=150, sequenceid=434, filesize=11.9 K 2023-03-24 16:54:27,493 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/d3124a90a6d04bc0b4d51640a343b504 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504 2023-03-24 16:54:27,498 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504, entries=150, sequenceid=434, filesize=11.9 K 2023-03-24 16:54:27,498 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 894ms, sequenceid=434, compaction requested=false 2023-03-24 16:54:27,499 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:27,499 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:27,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:27,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:27,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:27,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:27,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50208e06[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:27,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:27,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,506 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:27,506 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,506 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:27,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:27,511 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:27,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:27,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:27,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:27,511 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:27,513 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,513 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:27,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,513 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:27,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:27,513 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,514 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:27,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:27,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:27,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:27,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,517 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,517 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:27,518 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:27,517 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:27,618 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:27,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:27,621 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:27,621 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:27,621 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:27,621 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:27,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:27,623 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:27,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:27,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:27,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:27,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:27,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:27,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:27,628 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:27,628 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:27,629 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:27,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:27,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676927988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:27,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676927994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,053 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244b6b7b3309ac4cd6ab2a52194466dab7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b6b7b3309ac4cd6ab2a52194466dab7 2023-03-24 16:54:28,059 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=456, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e5fca510e1e5400ba27e32227592c813 2023-03-24 16:54:28,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:28,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676928291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:28,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676928297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,468 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/367359ed77594874b4cb82a825c523ca 2023-03-24 16:54:28,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:28,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676928794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676928803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,880 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/03dec0e1e45a485dbc4faa7c9367d773 2023-03-24 16:54:28,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e5fca510e1e5400ba27e32227592c813 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813 2023-03-24 16:54:28,888 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813, entries=150, sequenceid=456, filesize=25.4 K 2023-03-24 16:54:28,889 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/367359ed77594874b4cb82a825c523ca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca 2023-03-24 16:54:28,893 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca, entries=150, sequenceid=456, filesize=11.9 K 2023-03-24 16:54:28,894 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/03dec0e1e45a485dbc4faa7c9367d773 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773 2023-03-24 16:54:28,898 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773, entries=150, sequenceid=456, filesize=11.9 K 2023-03-24 16:54:28,899 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1271ms, sequenceid=456, compaction requested=true 2023-03-24 16:54:28,899 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:28,899 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:28,901 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f097861[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:28,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:28,903 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,904 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:28,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:28,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:28,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:28,912 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:28,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:28,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:28,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:28,912 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:28,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:28,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:28,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:28,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:28,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:28,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:28,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:28,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:28,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:28,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:28,918 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:28,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:28,918 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:29,018 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:29,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:29,022 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:29,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:29,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:29,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:29,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:29,024 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:29,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:29,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:29,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:29,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:29,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:29,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:29,026 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:29,027 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:29,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:29,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:29,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676929341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676929443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,454 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032455a2adc6ff454617b4d5796764d2cbfb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455a2adc6ff454617b4d5796764d2cbfb 2023-03-24 16:54:29,467 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=468, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e30eff9c1ccf47f49245c6f3e4e27b81 2023-03-24 16:54:29,488 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/77963b8dabde4c35b801712d3e4b1796 2023-03-24 16:54:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676929644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676929796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676929808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,900 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/409459ed68674870845e5c3ff9443b95 2023-03-24 16:54:29,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e30eff9c1ccf47f49245c6f3e4e27b81 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81 2023-03-24 16:54:29,909 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81, entries=150, sequenceid=468, filesize=25.4 K 2023-03-24 16:54:29,909 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/77963b8dabde4c35b801712d3e4b1796 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796 2023-03-24 16:54:29,913 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796, entries=150, sequenceid=468, filesize=11.9 K 2023-03-24 16:54:29,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/409459ed68674870845e5c3ff9443b95 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95 2023-03-24 16:54:29,917 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95, entries=150, sequenceid=468, filesize=11.9 K 2023-03-24 16:54:29,918 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 892ms, sequenceid=468, compaction requested=true 2023-03-24 16:54:29,918 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:29,918 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:29,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:29,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,923 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:29,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4ce301[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:29,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:29,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:29,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:29,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:29,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:29,927 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:29,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:29,930 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:29,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:29,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:29,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:29,930 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:29,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:29,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:29,933 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:29,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:29,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:29,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:29,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:29,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:29,938 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:29,938 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:29,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:29,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:29,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:29,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:29,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:29,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:29,964 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032400a1d9eaa67a4cfbbbba35d2a1f77a08 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400a1d9eaa67a4cfbbbba35d2a1f77a08 2023-03-24 16:54:29,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:29,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676929989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,038 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,041 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,048 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c6894b5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,058 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,058 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,060 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,060 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,060 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,065 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,066 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676930091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,166 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,169 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,180 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86c6180[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,188 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,188 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,198 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,198 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,198 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,198 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,198 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,199 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:30,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,199 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:30,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676930293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,299 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,302 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,308 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,308 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc2ca9c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,315 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,315 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,317 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,318 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,320 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,320 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:30,320 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,374 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=489, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/8cd2558bf8a941ec851bb41155f15cfe 2023-03-24 16:54:30,420 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,424 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,427 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@750f68ee[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,433 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,445 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,445 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:30,445 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,546 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,552 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,556 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,556 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,556 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,556 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d5a97c6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,571 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,576 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,576 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,586 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,585 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,586 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:30,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676930597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,686 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,689 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,690 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,690 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,694 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,694 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,694 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:30,694 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:30,695 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d18b8c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:30,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:30,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,698 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:30,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:30,700 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:30,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:30,702 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:30,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:30,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:30,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:30,702 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:30,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:30,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:30,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:30,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:30,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:30,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:30,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:30,709 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,709 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:30,709 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:30,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/34b14418f36d407c8765548ac3166fbe 2023-03-24 16:54:30,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/5734f2b6943b4d94bf4295a78747f0c6 2023-03-24 16:54:30,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/8cd2558bf8a941ec851bb41155f15cfe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe 2023-03-24 16:54:30,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe, entries=200, sequenceid=489, filesize=32.3 K 2023-03-24 16:54:30,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/34b14418f36d407c8765548ac3166fbe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe 2023-03-24 16:54:30,809 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:30,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe, entries=150, sequenceid=489, filesize=11.9 K 2023-03-24 16:54:30,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:30,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/5734f2b6943b4d94bf4295a78747f0c6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6 2023-03-24 16:54:30,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:30,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:30,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:30,813 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:30,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:30,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:30,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:30,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:30,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6, entries=150, sequenceid=489, filesize=11.9 K 2023-03-24 16:54:30,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:30,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 865ms, sequenceid=489, compaction requested=true 2023-03-24 16:54:30,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:30,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:30,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:30,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:54:30,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 16:54:30,816 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:30,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:30,816 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:30,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:30,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:30,817 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:30,817 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:30,818 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:30,818 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/A is initiating minor compaction (all files) 2023-03-24 16:54:30,818 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/A in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,818 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4241d937eb5b4f4fab95f5093fddbfc2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=134.8 K 2023-03-24 16:54:30,819 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4241d937eb5b4f4fab95f5093fddbfc2, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864580 2023-03-24 16:54:30,819 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:30,819 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/C is initiating minor compaction (all files) 2023-03-24 16:54:30,819 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/C in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:30,819 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f1d2948f8cb3442cacd5986a93f66bae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=60.3 K 2023-03-24 16:54:30,819 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6d0d3c092c65484b95019f168b38617d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679676864889 2023-03-24 16:54:30,820 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f1d2948f8cb3442cacd5986a93f66bae, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864580 2023-03-24 16:54:30,820 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e5fca510e1e5400ba27e32227592c813, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679676867062 2023-03-24 16:54:30,821 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d3124a90a6d04bc0b4d51640a343b504, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679676864889 2023-03-24 16:54:30,821 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e30eff9c1ccf47f49245c6f3e4e27b81, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679676867682 2023-03-24 16:54:30,822 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 03dec0e1e45a485dbc4faa7c9367d773, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679676867062 2023-03-24 16:54:30,822 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 8cd2558bf8a941ec851bb41155f15cfe, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679676869332 2023-03-24 16:54:30,822 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 409459ed68674870845e5c3ff9443b95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679676867682 2023-03-24 16:54:30,823 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5734f2b6943b4d94bf4295a78747f0c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679676869332 2023-03-24 16:54:30,834 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#C#compaction#192 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:30,850 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#A#compaction#193 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:31,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:31,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676871153, max timestamp=1679676871162Num uniques 859; 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676871153, max timestamp=1679676871162Num uniques 859; 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676871153, max timestamp=1679676871162Num uniques 859; 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:31,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676931217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:31,235 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249fdcdae5e8344c359278b0761be092f3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249fdcdae5e8344c359278b0761be092f3 2023-03-24 16:54:31,262 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f91eb0a447ad4405b9b7f3c84befd1ab as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f91eb0a447ad4405b9b7f3c84befd1ab 2023-03-24 16:54:31,266 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/C of e28bf4621cbd5e09c7a6e132bd2ee3a0 into f91eb0a447ad4405b9b7f3c84befd1ab(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:31,266 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:31,266 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/C, priority=11, startTime=1679676870816; duration=0sec 2023-03-24 16:54:31,267 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:31,267 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:54:31,276 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:54:31,276 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): e28bf4621cbd5e09c7a6e132bd2ee3a0/B is initiating minor compaction (all files) 2023-03-24 16:54:31,276 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e28bf4621cbd5e09c7a6e132bd2ee3a0/B in TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:31,276 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/21281d9ad18a4f0e941daebe88ba8903, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp, totalSize=60.3 K 2023-03-24 16:54:31,277 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 21281d9ad18a4f0e941daebe88ba8903, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679676864580 2023-03-24 16:54:31,277 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b9457d23598f491ba9bf89f158fb17de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679676864889 2023-03-24 16:54:31,277 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 367359ed77594874b4cb82a825c523ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679676867062 2023-03-24 16:54:31,278 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 77963b8dabde4c35b801712d3e4b1796, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679676867682 2023-03-24 16:54:31,278 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 34b14418f36d407c8765548ac3166fbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679676869332 2023-03-24 16:54:31,287 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): e28bf4621cbd5e09c7a6e132bd2ee3a0#B#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:31,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676931323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:31,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676931526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:31,640 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/b00d42b428c64bbd8b5bf12cd4e72e95 2023-03-24 16:54:31,651 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/5210675935fd43d3a7b9241e458aec95 2023-03-24 16:54:31,699 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/d168c8b337c3451cba0fa959dc1b055d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/d168c8b337c3451cba0fa959dc1b055d 2023-03-24 16:54:31,703 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/B of e28bf4621cbd5e09c7a6e132bd2ee3a0 into d168c8b337c3451cba0fa959dc1b055d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:31,703 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:31,703 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/B, priority=11, startTime=1679676870816; duration=0sec 2023-03-24 16:54:31,703 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:31,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47484 deadline: 1679676931810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:31,813 DEBUG [Thread-857] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:31,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47536 deadline: 1679676931817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:31,820 DEBUG [Thread-863] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:31,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:31,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676931827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,061 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f288bf9417e546debbb94bfbee42c16b 2023-03-24 16:54:32,065 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/b00d42b428c64bbd8b5bf12cd4e72e95 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/b00d42b428c64bbd8b5bf12cd4e72e95 2023-03-24 16:54:32,070 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/b00d42b428c64bbd8b5bf12cd4e72e95, entries=150, sequenceid=500, filesize=25.4 K 2023-03-24 16:54:32,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/5210675935fd43d3a7b9241e458aec95 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/5210675935fd43d3a7b9241e458aec95 2023-03-24 16:54:32,076 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/5210675935fd43d3a7b9241e458aec95, entries=150, sequenceid=500, filesize=11.9 K 2023-03-24 16:54:32,077 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/f288bf9417e546debbb94bfbee42c16b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f288bf9417e546debbb94bfbee42c16b 2023-03-24 16:54:32,082 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f288bf9417e546debbb94bfbee42c16b, entries=150, sequenceid=500, filesize=11.9 K 2023-03-24 16:54:32,083 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1266ms, sequenceid=500, compaction requested=false 2023-03-24 16:54:32,083 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:32,083 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:32,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,086 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:32,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:32,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbd37e0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:32,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:32,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,092 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:32,092 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:32,092 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:32,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:32,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,100 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/6269cd002c8e4e1f923093efe42ea4df as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6269cd002c8e4e1f923093efe42ea4df 2023-03-24 16:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:32,101 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:32,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:32,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:32,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:32,101 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:32,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:32,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,107 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e28bf4621cbd5e09c7a6e132bd2ee3a0/A of e28bf4621cbd5e09c7a6e132bd2ee3a0 into 6269cd002c8e4e1f923093efe42ea4df(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 16:54:32,107 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:32,107 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0., storeName=e28bf4621cbd5e09c7a6e132bd2ee3a0/A, priority=11, startTime=1679676870816; duration=1sec 2023-03-24 16:54:32,107 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:32,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:32,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:32,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:32,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,114 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:32,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:32,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,114 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:32,214 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:32,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:32,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:32,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:32,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:32,217 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:32,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:32,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:32,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:32,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:32,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,220 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:32,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:32,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:32,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:32,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. started... 2023-03-24 16:54:32,222 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:32,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:32,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. as already flushing 2023-03-24 16:54:32,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676932268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676932269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,329 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:62195 2023-03-24 16:54:32,329 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:32,330 DEBUG [Thread-868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:62195 2023-03-24 16:54:32,330 DEBUG [Thread-868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:32,330 DEBUG [Thread-874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:62195 2023-03-24 16:54:32,330 DEBUG [Thread-874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:32,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676932333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,339 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:62195 2023-03-24 16:54:32,339 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:32,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676932371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676932373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676932574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676932575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,638 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248c3a5f5a1f8a467d83afcc40522d993d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248c3a5f5a1f8a467d83afcc40522d993d 2023-03-24 16:54:32,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676932877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:32,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676932879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,043 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=539, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e65daefbbe7141fdbe3629096b893034 2023-03-24 16:54:33,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:33,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47542 deadline: 1679676933343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:33,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47478 deadline: 1679676933378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:33,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47524 deadline: 1679676933383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,454 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/86d75647c80141e6a7c7bc8d96991370 2023-03-24 16:54:33,863 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/d96fe3ed10a94a999077438a60326084 2023-03-24 16:54:33,867 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/e65daefbbe7141fdbe3629096b893034 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e65daefbbe7141fdbe3629096b893034 2023-03-24 16:54:33,870 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e65daefbbe7141fdbe3629096b893034, entries=150, sequenceid=539, filesize=25.4 K 2023-03-24 16:54:33,871 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/86d75647c80141e6a7c7bc8d96991370 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/86d75647c80141e6a7c7bc8d96991370 2023-03-24 16:54:33,874 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/86d75647c80141e6a7c7bc8d96991370, entries=150, sequenceid=539, filesize=11.9 K 2023-03-24 16:54:33,874 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/d96fe3ed10a94a999077438a60326084 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d96fe3ed10a94a999077438a60326084 2023-03-24 16:54:33,877 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d96fe3ed10a94a999077438a60326084, entries=150, sequenceid=539, filesize=11.9 K 2023-03-24 16:54:33,878 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1656ms, sequenceid=539, compaction requested=true 2023-03-24 16:54:33,878 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:33,878 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:33,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:33,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:33,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:33,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72fb7881[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:33,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:33,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:33,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:33,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:33,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:33,884 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:33,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:33,886 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:33,886 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:33,886 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:33,886 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:33,886 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:33,889 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,889 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:33,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,889 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:33,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,890 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:33,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:33,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:33,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:33,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:33,896 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:33,896 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:33,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:33,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:33,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:33,897 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,897 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:33,897 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:33,996 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:33,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:34,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:34,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:54:34,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:34,382 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:62195 2023-03-24 16:54:34,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:34,382 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:34,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:34,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:34,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:34,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:34,386 DEBUG [Thread-859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:62195 2023-03-24 16:54:34,386 DEBUG [Thread-859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:34,794 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324998aead0465f4a378a0a74ee6d291ede to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324998aead0465f4a378a0a74ee6d291ede 2023-03-24 16:54:35,200 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/ff6254770bad4af58b3b98658f46c869 2023-03-24 16:54:35,356 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:62195 2023-03-24 16:54:35,356 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:35,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/c67d3d2c8fbb47c9961e32b6c3b46f88 2023-03-24 16:54:35,840 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:62195 2023-03-24 16:54:35,840 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:35,854 DEBUG [Thread-857] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x399105f2 to 127.0.0.1:62195 2023-03-24 16:54:35,854 DEBUG [Thread-857] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 5192 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 5220 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 2240 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 6717 rows 2023-03-24 16:54:35,854 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 2249 2023-03-24 16:54:35,855 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 6740 rows 2023-03-24 16:54:35,855 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:54:35,855 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a607873 to 127.0.0.1:62195 2023-03-24 16:54:35,855 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:54:35,858 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:54:35,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:54:35,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:35,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:35,862 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676875862"}]},"ts":"1679676875862"} 2023-03-24 16:54:35,863 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:54:35,864 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:54:35,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, UNASSIGN}] 2023-03-24 16:54:35,866 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, UNASSIGN 2023-03-24 16:54:35,867 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:35,867 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676875867"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676875867"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676875867"}]},"ts":"1679676875867"} 2023-03-24 16:54:35,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:35,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:36,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:36,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e28bf4621cbd5e09c7a6e132bd2ee3a0, disabling compactions & flushes 2023-03-24 16:54:36,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:36,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/8a678be8f52649b99b42dcbc6468abe6 2023-03-24 16:54:36,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/ff6254770bad4af58b3b98658f46c869 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/ff6254770bad4af58b3b98658f46c869 2023-03-24 16:54:36,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/ff6254770bad4af58b3b98658f46c869, entries=150, sequenceid=551, filesize=25.4 K 2023-03-24 16:54:36,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/c67d3d2c8fbb47c9961e32b6c3b46f88 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c67d3d2c8fbb47c9961e32b6c3b46f88 2023-03-24 16:54:36,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c67d3d2c8fbb47c9961e32b6c3b46f88, entries=150, sequenceid=551, filesize=11.9 K 2023-03-24 16:54:36,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/8a678be8f52649b99b42dcbc6468abe6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/8a678be8f52649b99b42dcbc6468abe6 2023-03-24 16:54:36,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/8a678be8f52649b99b42dcbc6468abe6, entries=150, sequenceid=551, filesize=11.9 K 2023-03-24 16:54:36,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1654ms, sequenceid=551, compaction requested=true 2023-03-24 16:54:36,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:36,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:36,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:36,036 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. because compaction request was cancelled 2023-03-24 16:54:36,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:36,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:36,036 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. because compaction request was cancelled 2023-03-24 16:54:36,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:36,036 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. because compaction request was cancelled 2023-03-24 16:54:36,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. after waiting 0 ms 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:36,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e28bf4621cbd5e09c7a6e132bd2ee3a0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=A 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=B 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e28bf4621cbd5e09c7a6e132bd2ee3a0, store=C 2023-03-24 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:36,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244b68435c01824bd0b0d9044a69beed14 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b68435c01824bd0b0d9044a69beed14 2023-03-24 16:54:36,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:36,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=558, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/4ea383da7490417c9c4456a8c32230ac 2023-03-24 16:54:36,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:37,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/51df23ba44df42d7bc36c4e6fd02ddd5 2023-03-24 16:54:37,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/673171de51eb49b2a8ce5bd427d2a3cc 2023-03-24 16:54:37,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/A/4ea383da7490417c9c4456a8c32230ac as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4ea383da7490417c9c4456a8c32230ac 2023-03-24 16:54:37,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4ea383da7490417c9c4456a8c32230ac, entries=150, sequenceid=558, filesize=25.4 K 2023-03-24 16:54:37,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/B/51df23ba44df42d7bc36c4e6fd02ddd5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/51df23ba44df42d7bc36c4e6fd02ddd5 2023-03-24 16:54:37,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/51df23ba44df42d7bc36c4e6fd02ddd5, entries=150, sequenceid=558, filesize=11.9 K 2023-03-24 16:54:37,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/.tmp/C/673171de51eb49b2a8ce5bd427d2a3cc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/673171de51eb49b2a8ce5bd427d2a3cc 2023-03-24 16:54:37,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/673171de51eb49b2a8ce5bd427d2a3cc, entries=150, sequenceid=558, filesize=11.9 K 2023-03-24 16:54:37,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e28bf4621cbd5e09c7a6e132bd2ee3a0 in 1652ms, sequenceid=558, compaction requested=true 2023-03-24 16:54:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/f2af4f61314840cbb4bca7803c215fd1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/28368c01b1b24e76899b6708d9598eea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/78dc56b886674b76b2e026f24bea8419, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/1b491398dfbb4f9bbd1afc79efc88389, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2ffac5d61df14c3fbaf61073ecea6552, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4241d937eb5b4f4fab95f5093fddbfc2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe] to archive 2023-03-24 16:54:37,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:37,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/688a51ede2944a07821177154602076e 2023-03-24 16:54:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/cf4eea1acca64cceaa70f257c4dcd3d4 2023-03-24 16:54:37,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5dde15ad3c1e46b2b7a7e440e634454c 2023-03-24 16:54:37,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4cfa91dc8f1b47568baa8933fbe04c3e 2023-03-24 16:54:37,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/f2af4f61314840cbb4bca7803c215fd1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/f2af4f61314840cbb4bca7803c215fd1 2023-03-24 16:54:37,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/d6e825bc21c74e89b10d4b42af0d1120 2023-03-24 16:54:37,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5526ac657bf44d2eb56f390887babf13 2023-03-24 16:54:37,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5f0722ae46234aaea2ed579ed9dece11 2023-03-24 16:54:37,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/28368c01b1b24e76899b6708d9598eea to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/28368c01b1b24e76899b6708d9598eea 2023-03-24 16:54:37,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/a6842373ff2d48ef9e62ed9dedfeef66 2023-03-24 16:54:37,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/5c58684f64c141e5a2f314c5188ea711 2023-03-24 16:54:37,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/78dc56b886674b76b2e026f24bea8419 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/78dc56b886674b76b2e026f24bea8419 2023-03-24 16:54:37,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/c52e405c45ad4aa688164f9b3025e830 2023-03-24 16:54:37,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/1b491398dfbb4f9bbd1afc79efc88389 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/1b491398dfbb4f9bbd1afc79efc88389 2023-03-24 16:54:37,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6c1d30016e6c4cb88f7b06b350673f05 2023-03-24 16:54:37,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/7b396544a281438ca72c56ca6683a855 2023-03-24 16:54:37,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2d2eefe9ad2647039c8efbbb132250e7 2023-03-24 16:54:37,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/eec0bf3313fa4d1e88710d2cfad10ae0 2023-03-24 16:54:37,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2ffac5d61df14c3fbaf61073ecea6552 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/2ffac5d61df14c3fbaf61073ecea6552 2023-03-24 16:54:37,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9382032001ee4f738a3a80a4e3244b8e 2023-03-24 16:54:37,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/25401791689c47a1b3c26cf79347f7e8 2023-03-24 16:54:37,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/9ab225344a4c4309bf5b0750710ff22a 2023-03-24 16:54:37,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/799029eda5f2410ea47365cd9b91d27f 2023-03-24 16:54:37,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4241d937eb5b4f4fab95f5093fddbfc2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4241d937eb5b4f4fab95f5093fddbfc2 2023-03-24 16:54:37,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6d0d3c092c65484b95019f168b38617d 2023-03-24 16:54:37,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e5fca510e1e5400ba27e32227592c813 2023-03-24 16:54:37,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e30eff9c1ccf47f49245c6f3e4e27b81 2023-03-24 16:54:37,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/8cd2558bf8a941ec851bb41155f15cfe 2023-03-24 16:54:37,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8a419171470a4f9ab3f8eb25da119fe6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b5c8beed71fd43788c98f9b8d47fd330, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/efe30d95bab4480bb7befab946df0a9b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/610257b9c15346f78bd7701e53572513, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/a941c2c1e27c4d28aa05ce1d0a4f5d86, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/21281d9ad18a4f0e941daebe88ba8903, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe] to archive 2023-03-24 16:54:37,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:37,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/498d22dfefab4bbebe1465a234ae5fdd 2023-03-24 16:54:37,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/cae327c5d12446669f3a953d4011a248 2023-03-24 16:54:37,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/eec16b83f43c4e739d87c55ab8e83d1b 2023-03-24 16:54:37,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/045fc32ca6614b1aa787ec9ab3e7c328 2023-03-24 16:54:37,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8a419171470a4f9ab3f8eb25da119fe6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8a419171470a4f9ab3f8eb25da119fe6 2023-03-24 16:54:37,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c2f9f0c8dfd44e4ab41266f3c220630b 2023-03-24 16:54:37,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/1578f7410a0f4dd5a344ee1d10db5990 2023-03-24 16:54:37,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b5c8beed71fd43788c98f9b8d47fd330 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b5c8beed71fd43788c98f9b8d47fd330 2023-03-24 16:54:37,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/6fe9a6f39d6b4bd8aa892de15674586f 2023-03-24 16:54:37,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/f815d20728c5484d8791b074675a5dbb 2023-03-24 16:54:37,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/efe30d95bab4480bb7befab946df0a9b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/efe30d95bab4480bb7befab946df0a9b 2023-03-24 16:54:37,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/8483d8ffe5954b2299a039a9f4eef017 2023-03-24 16:54:37,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/10d4c66008134c6eb08318033bcfbe5b 2023-03-24 16:54:37,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/610257b9c15346f78bd7701e53572513 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/610257b9c15346f78bd7701e53572513 2023-03-24 16:54:37,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e5a2efea41a44519ac60e209a09b1864 2023-03-24 16:54:37,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/115a287367694dd19f92d9aff7432718 2023-03-24 16:54:37,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b2a4805dcc194a6b824205e3beca1a38 2023-03-24 16:54:37,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/186ddcceaf2b4f1cbe3b3cf3b51fa85d 2023-03-24 16:54:37,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/a941c2c1e27c4d28aa05ce1d0a4f5d86 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/a941c2c1e27c4d28aa05ce1d0a4f5d86 2023-03-24 16:54:37,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/0e5f0dc7c3394b5b81b393282f5c110e 2023-03-24 16:54:37,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/fa94f90683eb40e1b28ca25bc925fbe2 2023-03-24 16:54:37,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/e1393c4c13254e0eaa780bef5c347e2b 2023-03-24 16:54:37,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/21281d9ad18a4f0e941daebe88ba8903 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/21281d9ad18a4f0e941daebe88ba8903 2023-03-24 16:54:37,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/7c38850b78434498bdbdb5534b0b6bdd 2023-03-24 16:54:37,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/b9457d23598f491ba9bf89f158fb17de 2023-03-24 16:54:37,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/367359ed77594874b4cb82a825c523ca 2023-03-24 16:54:37,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/77963b8dabde4c35b801712d3e4b1796 2023-03-24 16:54:37,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/34b14418f36d407c8765548ac3166fbe 2023-03-24 16:54:37,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/a4d8d41b3efe4e6299acc9a8d73e26b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/671b6a89d0f446c8acabc903de9e78c2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/dd9cf6d35979457bb282149250596e5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/9ac548d864f14b01a63cea95f628f8ff, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/042587c995694e378c4dda88570dd1ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f1d2948f8cb3442cacd5986a93f66bae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6] to archive 2023-03-24 16:54:37,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:54:37,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f8779a5a52fc4477bf4b74c1a07db058 2023-03-24 16:54:37,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/89728a6f95db434a877d5992abeb0e3e 2023-03-24 16:54:37,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/681399d05a18462cbc03d3bfd89881a9 2023-03-24 16:54:37,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f6e28101ede64af6aca40448d96544a7 2023-03-24 16:54:37,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/a4d8d41b3efe4e6299acc9a8d73e26b9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/a4d8d41b3efe4e6299acc9a8d73e26b9 2023-03-24 16:54:37,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/517e75a9c727454aac5671a8299423f2 2023-03-24 16:54:37,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e57810097bd747308e66f01d0f0f6aa5 2023-03-24 16:54:37,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/671b6a89d0f446c8acabc903de9e78c2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/671b6a89d0f446c8acabc903de9e78c2 2023-03-24 16:54:37,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/00a01be8c2584e43857b35810fe79226 2023-03-24 16:54:37,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/e057dc7d284a4549b6c5b20a0414c26e 2023-03-24 16:54:37,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/dd9cf6d35979457bb282149250596e5e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/dd9cf6d35979457bb282149250596e5e 2023-03-24 16:54:37,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/60c5ac01ba4b46528c857c9a4285ad78 2023-03-24 16:54:37,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/15906f456c5f4f8e94a3e54eddddd2e2 2023-03-24 16:54:37,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/9ac548d864f14b01a63cea95f628f8ff to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/9ac548d864f14b01a63cea95f628f8ff 2023-03-24 16:54:37,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/bf6444da0c8b499b973fb1343dd7feb2 2023-03-24 16:54:37,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/59c4fecc7f44471c940d6db59d970a64 2023-03-24 16:54:37,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/7b7ed8f58f8a46b49bb4fd54bef6374f 2023-03-24 16:54:37,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/fab81fa1df04470ab59a80466374b62b 2023-03-24 16:54:37,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/042587c995694e378c4dda88570dd1ef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/042587c995694e378c4dda88570dd1ef 2023-03-24 16:54:37,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/af3ab0e88d31482f91db322bf93d1846 2023-03-24 16:54:37,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/95d0ecc220c34bcf8bf29abe33dfabd1 2023-03-24 16:54:37,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/3490b5e91ab44247b1b16eef5ac2621e 2023-03-24 16:54:37,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f1d2948f8cb3442cacd5986a93f66bae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f1d2948f8cb3442cacd5986a93f66bae 2023-03-24 16:54:37,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/2305f15fa61f4f0bb66f6386bb2a37b7 2023-03-24 16:54:37,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d3124a90a6d04bc0b4d51640a343b504 2023-03-24 16:54:37,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/03dec0e1e45a485dbc4faa7c9367d773 2023-03-24 16:54:37,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/409459ed68674870845e5c3ff9443b95 2023-03-24 16:54:37,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/5734f2b6943b4d94bf4295a78747f0c6 2023-03-24 16:54:37,802 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits/561.seqid, newMaxSeqId=561, maxSeqId=4 2023-03-24 16:54:37,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0. 2023-03-24 16:54:37,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e28bf4621cbd5e09c7a6e132bd2ee3a0: 2023-03-24 16:54:37,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:37,805 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e28bf4621cbd5e09c7a6e132bd2ee3a0, regionState=CLOSED 2023-03-24 16:54:37,805 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676877805"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676877805"}]},"ts":"1679676877805"} 2023-03-24 16:54:37,807 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-24 16:54:37,807 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure e28bf4621cbd5e09c7a6e132bd2ee3a0, server=jenkins-hbase4.apache.org,33315,1679676819242 in 1.9390 sec 2023-03-24 16:54:37,809 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-24 16:54:37,809 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e28bf4621cbd5e09c7a6e132bd2ee3a0, UNASSIGN in 1.9420 sec 2023-03-24 16:54:37,809 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676877809"}]},"ts":"1679676877809"} 2023-03-24 16:54:37,810 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:54:37,811 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:54:37,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9540 sec 2023-03-24 16:54:37,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 16:54:37,966 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-24 16:54:37,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:54:37,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:37,969 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 16:54:37,969 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:37,973 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:37,975 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits] 2023-03-24 16:54:37,977 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4ea383da7490417c9c4456a8c32230ac to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/4ea383da7490417c9c4456a8c32230ac 2023-03-24 16:54:37,978 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6269cd002c8e4e1f923093efe42ea4df to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/6269cd002c8e4e1f923093efe42ea4df 2023-03-24 16:54:37,979 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/b00d42b428c64bbd8b5bf12cd4e72e95 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/b00d42b428c64bbd8b5bf12cd4e72e95 2023-03-24 16:54:37,980 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e65daefbbe7141fdbe3629096b893034 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/e65daefbbe7141fdbe3629096b893034 2023-03-24 16:54:37,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/ff6254770bad4af58b3b98658f46c869 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/A/ff6254770bad4af58b3b98658f46c869 2023-03-24 16:54:37,983 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/51df23ba44df42d7bc36c4e6fd02ddd5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/51df23ba44df42d7bc36c4e6fd02ddd5 2023-03-24 16:54:37,983 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/5210675935fd43d3a7b9241e458aec95 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/5210675935fd43d3a7b9241e458aec95 2023-03-24 16:54:37,984 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/86d75647c80141e6a7c7bc8d96991370 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/86d75647c80141e6a7c7bc8d96991370 2023-03-24 16:54:37,985 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c67d3d2c8fbb47c9961e32b6c3b46f88 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/c67d3d2c8fbb47c9961e32b6c3b46f88 2023-03-24 16:54:37,986 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/d168c8b337c3451cba0fa959dc1b055d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/B/d168c8b337c3451cba0fa959dc1b055d 2023-03-24 16:54:37,988 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/673171de51eb49b2a8ce5bd427d2a3cc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/673171de51eb49b2a8ce5bd427d2a3cc 2023-03-24 16:54:37,989 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/8a678be8f52649b99b42dcbc6468abe6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/8a678be8f52649b99b42dcbc6468abe6 2023-03-24 16:54:37,990 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d96fe3ed10a94a999077438a60326084 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/d96fe3ed10a94a999077438a60326084 2023-03-24 16:54:37,991 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f288bf9417e546debbb94bfbee42c16b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f288bf9417e546debbb94bfbee42c16b 2023-03-24 16:54:37,992 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f91eb0a447ad4405b9b7f3c84befd1ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/C/f91eb0a447ad4405b9b7f3c84befd1ab 2023-03-24 16:54:37,994 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits/561.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0/recovered.edits/561.seqid 2023-03-24 16:54:37,995 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/e28bf4621cbd5e09c7a6e132bd2ee3a0 2023-03-24 16:54:37,995 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:54:37,995 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:54:37,996 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 16:54:37,999 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400a1d9eaa67a4cfbbbba35d2a1f77a08 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400a1d9eaa67a4cfbbbba35d2a1f77a08 2023-03-24 16:54:38,000 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240356bec49bd64fbd8154ee2101ef6fb3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240356bec49bd64fbd8154ee2101ef6fb3 2023-03-24 16:54:38,001 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240f999d4aba08409ea678c8fa33bb703f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240f999d4aba08409ea678c8fa33bb703f 2023-03-24 16:54:38,002 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324249072d6ccd24191b5b422a650b8dbc2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324249072d6ccd24191b5b422a650b8dbc2 2023-03-24 16:54:38,003 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032426ce549d61f54988a7382a152d74c22a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032426ce549d61f54988a7382a152d74c22a 2023-03-24 16:54:38,004 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324280208318f0b49a8aeb34bafc1dcf59d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324280208318f0b49a8aeb34bafc1dcf59d 2023-03-24 16:54:38,005 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c3b1ad3864f4ad1a26a24bb858dc7d9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c3b1ad3864f4ad1a26a24bb858dc7d9 2023-03-24 16:54:38,006 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243b33768c7104475c8a50a2765578e574 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243b33768c7104475c8a50a2765578e574 2023-03-24 16:54:38,007 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b68435c01824bd0b0d9044a69beed14 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b68435c01824bd0b0d9044a69beed14 2023-03-24 16:54:38,008 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b6b7b3309ac4cd6ab2a52194466dab7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b6b7b3309ac4cd6ab2a52194466dab7 2023-03-24 16:54:38,009 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032452dae45638404a07badec1aa704c7376 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032452dae45638404a07badec1aa704c7376 2023-03-24 16:54:38,010 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455310e7b42034a19a8d9ea3ad2be7d5f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455310e7b42034a19a8d9ea3ad2be7d5f 2023-03-24 16:54:38,011 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455a2adc6ff454617b4d5796764d2cbfb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032455a2adc6ff454617b4d5796764d2cbfb 2023-03-24 16:54:38,012 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032464a19edce4be483cb987c26d6744b2e1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032464a19edce4be483cb987c26d6744b2e1 2023-03-24 16:54:38,013 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324716116492f594aa08b50291b60e3827d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324716116492f594aa08b50291b60e3827d 2023-03-24 16:54:38,014 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247a512cad786d4372b009643ddb605230 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247a512cad786d4372b009643ddb605230 2023-03-24 16:54:38,015 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482a8812bbdbe4c13bd3b2fed72ade51d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482a8812bbdbe4c13bd3b2fed72ade51d 2023-03-24 16:54:38,016 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ad573ac53194d93aae47423241ea293 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ad573ac53194d93aae47423241ea293 2023-03-24 16:54:38,017 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248b6e9ee4944d4eccad7257350460933d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248b6e9ee4944d4eccad7257350460933d 2023-03-24 16:54:38,018 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248c3a5f5a1f8a467d83afcc40522d993d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248c3a5f5a1f8a467d83afcc40522d993d 2023-03-24 16:54:38,020 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324998aead0465f4a378a0a74ee6d291ede to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324998aead0465f4a378a0a74ee6d291ede 2023-03-24 16:54:38,021 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249fdcdae5e8344c359278b0761be092f3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249fdcdae5e8344c359278b0761be092f3 2023-03-24 16:54:38,022 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a04e8bb90f1f4c1085a99151312759d7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a04e8bb90f1f4c1085a99151312759d7 2023-03-24 16:54:38,023 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c6aff01f2dbb4527838fefbbcf529e25 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c6aff01f2dbb4527838fefbbcf529e25 2023-03-24 16:54:38,024 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c9822ffd91cc4dc1a95f6c07fab7ee3c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c9822ffd91cc4dc1a95f6c07fab7ee3c 2023-03-24 16:54:38,025 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fe8564dab16a40d5a4bb4037518f2634 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fe8564dab16a40d5a4bb4037518f2634 2023-03-24 16:54:38,026 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:54:38,029 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:38,032 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:54:38,034 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:54:38,035 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:38,035 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:54:38,035 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676878035"}]},"ts":"9223372036854775807"} 2023-03-24 16:54:38,037 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:54:38,037 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e28bf4621cbd5e09c7a6e132bd2ee3a0, NAME => 'TestAcidGuarantees,,1679676849190.e28bf4621cbd5e09c7a6e132bd2ee3a0.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:54:38,037 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:54:38,037 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676878037"}]},"ts":"9223372036854775807"} 2023-03-24 16:54:38,047 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:54:38,049 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:38,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 82 msec 2023-03-24 16:54:38,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 16:54:38,070 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-24 16:54:38,083 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 234), OpenFileDescriptor=540 (was 572), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=366 (was 401), ProcessCount=170 (was 170), AvailableMemoryMB=5698 (was 5901) 2023-03-24 16:54:38,094 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=540, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=170, AvailableMemoryMB=5698 2023-03-24 16:54:38,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:54:38,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:54:38,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:54:38,098 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:54:38,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-24 16:54:38,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:38,099 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:54:38,101 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:38,101 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b empty. 2023-03-24 16:54:38,101 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:38,101 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:54:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:38,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:38,509 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:54:38,511 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7d260a92475e999cfa4ef78eea03ae7b, NAME => 'TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:54:38,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:38,916 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:54:38,916 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7d260a92475e999cfa4ef78eea03ae7b, disabling compactions & flushes 2023-03-24 16:54:38,916 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:38,916 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:38,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. after waiting 0 ms 2023-03-24 16:54:38,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:38,917 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:38,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:38,918 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:54:38,919 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676878919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676878919"}]},"ts":"1679676878919"} 2023-03-24 16:54:38,920 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:54:38,921 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:54:38,921 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676878921"}]},"ts":"1679676878921"} 2023-03-24 16:54:38,922 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:54:38,924 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, ASSIGN}] 2023-03-24 16:54:38,926 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, ASSIGN 2023-03-24 16:54:38,926 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:54:39,078 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=7d260a92475e999cfa4ef78eea03ae7b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:39,078 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676879078"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676879078"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676879078"}]},"ts":"1679676879078"} 2023-03-24 16:54:39,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:54:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:39,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7d260a92475e999cfa4ef78eea03ae7b, NAME => 'TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:54:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:54:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,236 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,237 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A 2023-03-24 16:54:39,237 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A 2023-03-24 16:54:39,237 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:39,238 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d260a92475e999cfa4ef78eea03ae7b columnFamilyName A 2023-03-24 16:54:39,238 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(310): Store=7d260a92475e999cfa4ef78eea03ae7b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:39,238 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,239 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B 2023-03-24 16:54:39,239 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B 2023-03-24 16:54:39,239 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:39,240 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d260a92475e999cfa4ef78eea03ae7b columnFamilyName B 2023-03-24 16:54:39,240 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(310): Store=7d260a92475e999cfa4ef78eea03ae7b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:39,240 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,241 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C 2023-03-24 16:54:39,241 DEBUG [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C 2023-03-24 16:54:39,241 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:54:39,241 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d260a92475e999cfa4ef78eea03ae7b columnFamilyName C 2023-03-24 16:54:39,242 INFO [StoreOpener-7d260a92475e999cfa4ef78eea03ae7b-1] regionserver.HStore(310): Store=7d260a92475e999cfa4ef78eea03ae7b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:54:39,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:39,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:54:39,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:39,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:54:39,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7d260a92475e999cfa4ef78eea03ae7b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68727963, jitterRate=0.024126455187797546}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:54:39,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:39,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., pid=30, masterSystemTime=1679676879231 2023-03-24 16:54:39,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:39,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:39,249 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=7d260a92475e999cfa4ef78eea03ae7b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:39,249 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676879249"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676879249"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676879249"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676879249"}]},"ts":"1679676879249"} 2023-03-24 16:54:39,251 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-24 16:54:39,251 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 in 171 msec 2023-03-24 16:54:39,252 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-24 16:54:39,253 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, ASSIGN in 327 msec 2023-03-24 16:54:39,253 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:54:39,253 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676879253"}]},"ts":"1679676879253"} 2023-03-24 16:54:39,254 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:54:39,257 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:54:39,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-03-24 16:54:40,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 16:54:40,204 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-24 16:54:40,205 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x112921f2 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,210 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22b4eae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,211 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,212 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,213 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:54:40,215 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44694, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:54:40,216 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,222 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,223 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,226 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,227 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,231 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,232 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,235 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,235 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,238 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,239 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,245 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,246 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,250 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,251 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,254 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,254 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,257 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,258 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:54:40,261 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:54:40,265 DEBUG [hconnection-0x510aa473-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:40,265 DEBUG [hconnection-0x7c6c3b2d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,269 DEBUG [hconnection-0x4cd46def-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,269 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39222, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,269 DEBUG [hconnection-0x472e01b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,269 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39206, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,269 DEBUG [hconnection-0x2fa75ab9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,270 DEBUG [hconnection-0x63d876cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,270 DEBUG [hconnection-0x7c38e080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,272 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39232, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,272 DEBUG [hconnection-0x52ebddbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:40,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:40,273 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:40,273 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:40,276 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,274 DEBUG [hconnection-0xc416527-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,277 DEBUG [hconnection-0x4d950b0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:54:40,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:40,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:40,278 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,280 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,280 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,281 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39304, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,281 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39246, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:40,282 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:54:40,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:40,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:40,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:40,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:40,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:40,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:40,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:40,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:40,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:40,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:40,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:40,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:40,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:40,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:40,293 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 16:54:40,294 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:40,294 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:40,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:40,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:40,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676880310, max timestamp=1679676880314Num uniques 859; 2023-03-24 16:54:40,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:40,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676880310, max timestamp=1679676880314Num uniques 859; 2023-03-24 16:54:40,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:40,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676880310, max timestamp=1679676880314Num uniques 859; 2023-03-24 16:54:40,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:40,327 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3fd5c413976a4274b1f460fc8c3fb327 2023-03-24 16:54:40,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676940330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676940331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676880325, max timestamp=1679676880331Num uniques 1018; 2023-03-24 16:54:40,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:54:40,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676880325, max timestamp=1679676880331Num uniques 1018; 2023-03-24 16:54:40,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:54:40,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676880327, max timestamp=1679676880331Num uniques 1018; 2023-03-24 16:54:40,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 16:54:40,358 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/18e253766a4840aa92f07a75d511c982 2023-03-24 16:54:40,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676940358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676880356, max timestamp=1679676880361Num uniques 1077; 2023-03-24 16:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676880356, max timestamp=1679676880361Num uniques 1077; 2023-03-24 16:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676880356, max timestamp=1679676880361Num uniques 1009; 2023-03-24 16:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-03-24 16:54:40,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676940403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676940404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676940436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676940436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676940465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676940507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676940507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676940638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676940640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676940668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676940709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676940710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,838 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/7b984f38fe7f40a2a779c000128d9d68 2023-03-24 16:54:40,843 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3fd5c413976a4274b1f460fc8c3fb327 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327 2023-03-24 16:54:40,847 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327, entries=100, sequenceid=10, filesize=9.3 K 2023-03-24 16:54:40,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/18e253766a4840aa92f07a75d511c982 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982 2023-03-24 16:54:40,853 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982, entries=100, sequenceid=10, filesize=9.3 K 2023-03-24 16:54:40,854 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/7b984f38fe7f40a2a779c000128d9d68 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68 2023-03-24 16:54:40,859 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68, entries=100, sequenceid=10, filesize=9.3 K 2023-03-24 16:54:40,860 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7d260a92475e999cfa4ef78eea03ae7b in 567ms, sequenceid=10, compaction requested=false 2023-03-24 16:54:40,860 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:40,860 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:40,860 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:40,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:40,862 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:40,863 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,863 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2625bf[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:40,864 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:40,864 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,866 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:40,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:40,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:40,866 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,866 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:40,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:40,872 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:40,872 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:40,872 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:40,872 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:40,872 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:40,874 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,874 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,874 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:40,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:40,875 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,876 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:40,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:40,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:40,896 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:40,896 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:40,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:40,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:40,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:40,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676880398, max timestamp=1679676880941Num uniques 1086; 2023-03-24 16:54:40,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-24 16:54:40,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676880398, max timestamp=1679676880941Num uniques 1086; 2023-03-24 16:54:40,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-24 16:54:40,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=107.34 KB heapSize=300.67 KB 2023-03-24 16:54:40,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676880398, max timestamp=1679676880941Num uniques 1086; 2023-03-24 16:54:40,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-24 16:54:40,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:40,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:40,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:40,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:40,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:40,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:40,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:40,957 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 16:54:40,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/55f0a71f8ae2439ab4c9b1693cc9643a 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676880955, max timestamp=1679676880969Num uniques 859; 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676880955, max timestamp=1679676880969Num uniques 859; 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676880955, max timestamp=1679676880969Num uniques 859; 2023-03-24 16:54:40,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:40,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/779db1695e614e548e73a9d4496c47f0 2023-03-24 16:54:40,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676880986, max timestamp=1679676880993Num uniques 1068; 2023-03-24 16:54:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 16:54:40,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676880986, max timestamp=1679676880993Num uniques 1068; 2023-03-24 16:54:40,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676880986, max timestamp=1679676880993Num uniques 1068; 2023-03-24 16:54:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 16:54:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 16:54:40,996 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:40,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:40,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,013 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,014 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,014 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,015 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676881005, max timestamp=1679676881013Num uniques 1077; 2023-03-24 16:54:41,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:54:41,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676881005, max timestamp=1679676881013Num uniques 1077; 2023-03-24 16:54:41,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679676881005, max timestamp=1679676881013Num uniques 1077; 2023-03-24 16:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:54:41,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,018 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-24 16:54:41,018 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,018 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,018 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,021 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,022 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41215abc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,025 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,026 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,026 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,027 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,029 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,030 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,030 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,030 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,030 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,031 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,031 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,031 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676881023, max timestamp=1679676881034Num uniques 1086; 2023-03-24 16:54:41,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-03-24 16:54:41,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676881023, max timestamp=1679676881034Num uniques 1136; 2023-03-24 16:54:41,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 16:54:41,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676881023, max timestamp=1679676881034Num uniques 1086; 2023-03-24 16:54:41,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-24 16:54:41,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,040 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,040 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676881045, max timestamp=1679676881049Num uniques 1045; 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676881045, max timestamp=1679676881049Num uniques 1045; 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676881045, max timestamp=1679676881049Num uniques 1009; 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 16:54:41,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676881087, max timestamp=1679676881090Num uniques 1104; 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676881087, max timestamp=1679676881090Num uniques 1104; 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679676881087, max timestamp=1679676881090Num uniques 1104; 2023-03-24 16:54:41,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-24 16:54:41,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676941090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676881101, max timestamp=1679676881131Num uniques 1063; 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676881101, max timestamp=1679676881131Num uniques 1063; 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676881101, max timestamp=1679676881131Num uniques 1063; 2023-03-24 16:54:41,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-24 16:54:41,140 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,147 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,149 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676881142, max timestamp=1679676881151Num uniques 1022; 2023-03-24 16:54:41,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676881142, max timestamp=1679676881151Num uniques 1022; 2023-03-24 16:54:41,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-24 16:54:41,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676881142, max timestamp=1679676881151Num uniques 1022; 2023-03-24 16:54:41,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-24 16:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-24 16:54:41,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5436674f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,158 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,166 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,166 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,175 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,175 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679676881160, max timestamp=1679676881174Num uniques 1081; 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679676881160, max timestamp=1679676881174Num uniques 1081; 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679676881160, max timestamp=1679676881174Num uniques 1081; 2023-03-24 16:54:41,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-24 16:54:41,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679676881185, max timestamp=1679676881191Num uniques 1140; 2023-03-24 16:54:41,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-24 16:54:41,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679676881185, max timestamp=1679676881191Num uniques 1140; 2023-03-24 16:54:41,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-24 16:54:41,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679676881185, max timestamp=1679676881191Num uniques 1140; 2023-03-24 16:54:41,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-24 16:54:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676881201, max timestamp=1679676881207Num uniques 1099; 2023-03-24 16:54:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-24 16:54:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676881201, max timestamp=1679676881207Num uniques 1099; 2023-03-24 16:54:41,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-24 16:54:41,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676881201, max timestamp=1679676881207Num uniques 1099; 2023-03-24 16:54:41,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679676881220, max timestamp=1679676881228Num uniques 1058; 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679676881220, max timestamp=1679676881228Num uniques 1058; 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=11808, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=11808, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679676881220, max timestamp=1679676881228Num uniques 1058; 2023-03-24 16:54:41,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=11808, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679676881238, max timestamp=1679676881243Num uniques 1117; 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679676881238, max timestamp=1679676881243Num uniques 1117; 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679676881238, max timestamp=1679676881243Num uniques 1117; 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 16:54:41,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 16:54:41,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676941256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679676881254, max timestamp=1679676881259Num uniques 1126; 2023-03-24 16:54:41,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-03-24 16:54:41,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679676881254, max timestamp=1679676881259Num uniques 1126; 2023-03-24 16:54:41,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679676881254, max timestamp=1679676881259Num uniques 1126; 2023-03-24 16:54:41,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-03-24 16:54:41,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-03-24 16:54:41,275 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679676881271, max timestamp=1679676881274Num uniques 1085; 2023-03-24 16:54:41,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-24 16:54:41,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679676881271, max timestamp=1679676881274Num uniques 1085; 2023-03-24 16:54:41,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-24 16:54:41,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679676881271, max timestamp=1679676881274Num uniques 1085; 2023-03-24 16:54:41,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-24 16:54:41,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,281 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,285 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d202238[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,293 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,295 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,298 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,298 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679676881291, max timestamp=1679676881298Num uniques 1094; 2023-03-24 16:54:41,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679676881291, max timestamp=1679676881298Num uniques 1094; 2023-03-24 16:54:41,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 16:54:41,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 16:54:41,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679676881291, max timestamp=1679676881298Num uniques 1094; 2023-03-24 16:54:41,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 16:54:41,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,307 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,307 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679676881314, max timestamp=1679676881318Num uniques 1053; 2023-03-24 16:54:41,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 16:54:41,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679676881314, max timestamp=1679676881318Num uniques 1053; 2023-03-24 16:54:41,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 16:54:41,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679676881314, max timestamp=1679676881318Num uniques 1053; 2023-03-24 16:54:41,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679676881360, max timestamp=1679676881363Num uniques 1012; 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679676881360, max timestamp=1679676881363Num uniques 1012; 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679676881360, max timestamp=1679676881363Num uniques 1012; 2023-03-24 16:54:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679676881370, max timestamp=1679676881377Num uniques 1021; 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679676881370, max timestamp=1679676881377Num uniques 1021; 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679676881370, max timestamp=1679676881377Num uniques 1021; 2023-03-24 16:54:41,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-24 16:54:41,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676941391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679676881389, max timestamp=1679676881393Num uniques 1130; 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679676881389, max timestamp=1679676881393Num uniques 1130; 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679676881389, max timestamp=1679676881393Num uniques 1130; 2023-03-24 16:54:41,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-03-24 16:54:41,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/ba9487e805ab4b62bf33965ff19eabee 2023-03-24 16:54:41,408 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,412 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/55f0a71f8ae2439ab4c9b1693cc9643a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a 2023-03-24 16:54:41,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,414 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10934, totalHeapSize=25096, min timestamp=1679676881398, max timestamp=1679676881414Num uniques 1089; 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10934, totalHeapSize=25096, min timestamp=1679676881398, max timestamp=1679676881414Num uniques 1089; 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=10934, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=10934, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1679676881398, max timestamp=1679676881414Num uniques 1089; 2023-03-24 16:54:41,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 16:54:41,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=8920, suffix heap size=141224,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=195 2023-03-24 16:54:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/779db1695e614e548e73a9d4496c47f0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0 2023-03-24 16:54:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,418 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,419 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,419 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,419 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1fbfe[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 16:54:41,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/ba9487e805ab4b62bf33965ff19eabee as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee 2023-03-24 16:54:41,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 16:54:41,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,433 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,433 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.66 KB/65184, currentSize=65.12 KB/66686 for 7d260a92475e999cfa4ef78eea03ae7b in 492ms, sequenceid=82, compaction requested=false 2023-03-24 16:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:41,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=85.25 KB heapSize=233.56 KB 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:41,437 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,444 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,444 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676941453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676941454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676941457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676941457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676941496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,544 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,548 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6ea502[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676941558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676941558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676941561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676941561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,578 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,578 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,678 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,683 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,688 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,688 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,688 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,688 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,690 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b85afdf[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,697 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,699 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,699 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676941698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,706 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,707 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676941762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676941763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676941765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:41,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676941765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,807 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,811 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,819 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,819 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,819 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,819 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,824 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,824 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,824 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,824 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c87975[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,830 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,830 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,830 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,836 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,836 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,850 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,850 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e3154b84155b48e1b2629c72fe7de1fa 2023-03-24 16:54:41,950 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:41,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:41,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:41,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:41,955 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:41,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:41,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:41,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:41,960 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:41,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:41,973 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,973 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:41,973 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:41,973 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:41,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:41,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fce7679[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:41,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:41,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:41,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:41,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:41,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:41,991 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:41,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:41,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:41,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:41,991 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:41,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:41,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:41,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:41,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:41,997 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:41,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:41,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:41,997 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:41,997 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:41,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:42,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676942001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676942067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676942067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676942069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676942071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,097 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,116 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,119 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,119 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,119 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,119 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,132 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,132 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,132 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:42,132 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:42,133 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:42,135 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,135 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:42,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66318e37[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:42,140 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:42,140 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:42,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:42,144 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:42,148 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:42,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:42,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:42,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:42,148 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:42,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:42,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:42,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:42,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,162 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:42,162 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:42,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,262 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,268 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,270 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,271 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,271 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,276 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/359d1cdea49b4cc8b23b6f0b0cd6850e 2023-03-24 16:54:42,276 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,276 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:42,276 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:42,276 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:42,279 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:42,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60cc3e79[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:42,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:42,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,287 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:42,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:42,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:42,293 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:42,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:42,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:42,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:42,293 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:42,295 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,295 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,295 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,296 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:42,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:42,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,301 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:42,301 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:42,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,401 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,405 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,407 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:42,408 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:42,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:42,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:42,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:42,411 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,411 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,411 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,411 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74dbf0c4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:42,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:42,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,413 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:42,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:42,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:42,423 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:42,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:42,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:42,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:42,423 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:42,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:42,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,429 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:42,429 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:42,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676942504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,530 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,533 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,537 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,538 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:42,538 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:42,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:42,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:42,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:42,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588a2d98[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:42,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:42,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,542 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:42,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:42,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:42,542 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,542 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:42,543 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:42,546 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:42,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:42,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:42,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:42,546 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:42,548 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,548 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:42,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:42,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,553 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:42,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,553 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:42,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676942573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676942574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676942575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:42,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676942576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,653 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:42,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:42,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a04088[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:42,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:42,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:42,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:42,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:42,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:42,672 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:42,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:42,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:42,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:42,673 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:42,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:42,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:42,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:42,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:42,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:42,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:42,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,680 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:42,680 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:42,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:42,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:42,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=27.11 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/0436de402cc54c4491e5d1b18b756c22 2023-03-24 16:54:42,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e3154b84155b48e1b2629c72fe7de1fa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa 2023-03-24 16:54:42,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa, entries=250, sequenceid=454, filesize=16.7 K 2023-03-24 16:54:42,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/359d1cdea49b4cc8b23b6f0b0cd6850e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e 2023-03-24 16:54:42,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e, entries=150, sequenceid=454, filesize=11.9 K 2023-03-24 16:54:42,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/0436de402cc54c4491e5d1b18b756c22 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22 2023-03-24 16:54:42,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22, entries=150, sequenceid=454, filesize=11.9 K 2023-03-24 16:54:42,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~85.25 KB/87296, heapSize ~233.52 KB/239120, currentSize=93.93 KB/96180 for 7d260a92475e999cfa4ef78eea03ae7b in 1289ms, sequenceid=454, compaction requested=true 2023-03-24 16:54:42,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:42,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:42,726 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:42,726 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:42,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:42,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:42,727 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38494 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:42,727 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:42,727 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,727 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=37.6 K 2023-03-24 16:54:42,728 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:42,728 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3fd5c413976a4274b1f460fc8c3fb327, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679676880282 2023-03-24 16:54:42,728 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:42,728 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,728 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=32.8 K 2023-03-24 16:54:42,728 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 55f0a71f8ae2439ab4c9b1693cc9643a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676880398 2023-03-24 16:54:42,728 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 18e253766a4840aa92f07a75d511c982, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679676880282 2023-03-24 16:54:42,729 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e3154b84155b48e1b2629c72fe7de1fa, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881420 2023-03-24 16:54:42,729 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 779db1695e614e548e73a9d4496c47f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676880398 2023-03-24 16:54:42,729 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 359d1cdea49b4cc8b23b6f0b0cd6850e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881421 2023-03-24 16:54:42,741 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:42,743 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:42,762 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/325b758ff4d74db2b54d2389afacff7c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/325b758ff4d74db2b54d2389afacff7c 2023-03-24 16:54:42,767 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 325b758ff4d74db2b54d2389afacff7c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:42,767 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:42,767 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676882726; duration=0sec 2023-03-24 16:54:42,767 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:42,767 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:42,768 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:42,768 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:42,768 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,768 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=32.8 K 2023-03-24 16:54:42,769 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7b984f38fe7f40a2a779c000128d9d68, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679676880282 2023-03-24 16:54:42,769 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ba9487e805ab4b62bf33965ff19eabee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676880398 2023-03-24 16:54:42,770 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0436de402cc54c4491e5d1b18b756c22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881421 2023-03-24 16:54:42,780 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:42,781 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:42,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:42,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:42,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:42,792 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:42,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:42,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:42,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,794 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:42,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:42,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:42,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:42,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:42,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:42,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:42,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:42,828 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 16:54:42,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:42,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:42,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:42,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:42,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:42,829 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:42,851 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/c7de9a2c2f1544b0a35eb92b77a7e901 2023-03-24 16:54:42,874 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/264855925d9d4f7aa12c7c3228aeef98 2023-03-24 16:54:43,155 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/d78c507ddd0f4fcbbbe48c77b98ed1e2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d78c507ddd0f4fcbbbe48c77b98ed1e2 2023-03-24 16:54:43,161 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into d78c507ddd0f4fcbbbe48c77b98ed1e2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:43,161 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:43,161 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676882726; duration=0sec 2023-03-24 16:54:43,161 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:43,214 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/61d718a294624603816cb2a1b6da4455 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/61d718a294624603816cb2a1b6da4455 2023-03-24 16:54:43,219 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into 61d718a294624603816cb2a1b6da4455(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:43,220 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:43,220 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676882726; duration=0sec 2023-03-24 16:54:43,220 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:43,289 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/5e8cb50206bb4913a45a55e64c7a1f84 2023-03-24 16:54:43,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/c7de9a2c2f1544b0a35eb92b77a7e901 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901 2023-03-24 16:54:43,299 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901, entries=150, sequenceid=472, filesize=11.9 K 2023-03-24 16:54:43,300 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/264855925d9d4f7aa12c7c3228aeef98 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98 2023-03-24 16:54:43,305 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98, entries=150, sequenceid=472, filesize=11.9 K 2023-03-24 16:54:43,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/5e8cb50206bb4913a45a55e64c7a1f84 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84 2023-03-24 16:54:43,316 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84, entries=150, sequenceid=472, filesize=11.9 K 2023-03-24 16:54:43,317 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 489ms, sequenceid=472, compaction requested=false 2023-03-24 16:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@331e9f53[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,325 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,328 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,328 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,336 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:54:43,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,337 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,337 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,437 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:43,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:43,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:43,441 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:43,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:43,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:43,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:43,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:43,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:43,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18681424[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,450 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,450 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,450 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,453 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,453 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,463 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:43,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,463 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:43,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:43,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:43,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:43,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/27554dfc2e60469a95ea7c40a6aecd3d 2023-03-24 16:54:43,564 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:43,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:43,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:43,567 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:43,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:43,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:43,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:43,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676883546, max timestamp=1679676883568Num uniques 859; 2023-03-24 16:54:43,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:43,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676883546, max timestamp=1679676883568Num uniques 859; 2023-03-24 16:54:43,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:43,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676883546, max timestamp=1679676883568Num uniques 859; 2023-03-24 16:54:43,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:43,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:43,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:43,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:43,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,574 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11071c5c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,578 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,578 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,579 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,582 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,582 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,592 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,592 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,592 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676943593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39222 deadline: 1679676943593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676943595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676943595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39274 deadline: 1679676943596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,692 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:43,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:43,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:43,696 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:43,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:43,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:43,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39288 deadline: 1679676943697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,697 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676943697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:43,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:43,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:43,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:43,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:43,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:43,700 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,700 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:43,700 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:43,700 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676943699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676943699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2113aaf4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,704 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,711 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,714 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,714 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,722 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,723 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,723 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:43,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,823 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:43,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:43,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:43,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:43,826 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:43,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:43,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:43,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,828 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:43,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:43,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:43,830 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,831 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:43,831 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:43,831 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66304d82[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,842 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,842 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,844 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,846 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:43,849 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,849 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:43,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676943898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676943901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676943902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676943902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/708dfd3b552d409e88bbfecf6c4f1b1b 2023-03-24 16:54:43,949 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:43,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:43,952 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:43,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:43,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:43,953 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,953 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,954 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:43,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:43,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:43,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:43,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:43,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:43,957 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:43,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:43,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:43,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:43,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:43,959 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea1ac54[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:43,961 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:43,961 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:43,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:43,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:43,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:43,964 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:43,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:43,966 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:43,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:43,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:43,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:43,966 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:43,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:43,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:43,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,970 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:43,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:43,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:43,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:43,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:43,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:43,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:43,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:43,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:43,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:43,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:43,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:43,974 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:43,974 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:44,074 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:44,077 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:44,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:44,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:44,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:44,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:44,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:44,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:44,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:44,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:44,083 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17eda220[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:44,085 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:44,085 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:44,086 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:44,086 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:44,086 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:44,088 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:44,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,089 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:44,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:44,091 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:44,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:44,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:44,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:44,091 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:44,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,092 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,094 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:44,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,096 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:44,096 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,096 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:44,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,197 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:44,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:44,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:44,202 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:44,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:44,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:44,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676944202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676944204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676944206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:44,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:44,208 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,208 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:44,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676944207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,208 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:44,209 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,210 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:44,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b15ee23[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:44,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:44,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:44,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:44,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:44,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:44,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:44,217 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:44,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:44,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:44,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:44,218 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:44,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,222 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:44,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,226 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:44,226 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:44,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,326 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:44,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:44,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:44,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:44,330 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:44,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:44,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:44,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,331 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:44,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:44,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:44,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:44,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:44,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2dfa9e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:44,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:44,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,339 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:44,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,340 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,341 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:44,343 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:44,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:44,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:44,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:44,343 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:44,344 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,344 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,344 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,345 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:44,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:44,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,348 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:44,348 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:44,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/2e1d25d7551c40988a7ea13499e2a90b 2023-03-24 16:54:44,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/27554dfc2e60469a95ea7c40a6aecd3d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d 2023-03-24 16:54:44,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d, entries=150, sequenceid=485, filesize=11.9 K 2023-03-24 16:54:44,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/708dfd3b552d409e88bbfecf6c4f1b1b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b 2023-03-24 16:54:44,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b, entries=150, sequenceid=485, filesize=11.9 K 2023-03-24 16:54:44,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/2e1d25d7551c40988a7ea13499e2a90b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b 2023-03-24 16:54:44,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b, entries=150, sequenceid=485, filesize=11.9 K 2023-03-24 16:54:44,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7d260a92475e999cfa4ef78eea03ae7b in 862ms, sequenceid=485, compaction requested=true 2023-03-24 16:54:44,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:44,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:44,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:44,381 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:44,381 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:44,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:44,384 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:44,384 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:44,384 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,384 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:44,384 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/325b758ff4d74db2b54d2389afacff7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=35.8 K 2023-03-24 16:54:44,384 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:44,384 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,384 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d78c507ddd0f4fcbbbe48c77b98ed1e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=35.8 K 2023-03-24 16:54:44,384 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 325b758ff4d74db2b54d2389afacff7c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881421 2023-03-24 16:54:44,385 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d78c507ddd0f4fcbbbe48c77b98ed1e2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881421 2023-03-24 16:54:44,385 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c7de9a2c2f1544b0a35eb92b77a7e901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1679676881450 2023-03-24 16:54:44,385 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 264855925d9d4f7aa12c7c3228aeef98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1679676881450 2023-03-24 16:54:44,385 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 27554dfc2e60469a95ea7c40a6aecd3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:44,385 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 708dfd3b552d409e88bbfecf6c4f1b1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:44,391 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:44,391 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:44,448 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:44,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:44,452 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:44,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:44,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:44,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,456 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:44,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:44,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:44,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:44,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:44,459 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:44,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:44,470 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/f4b10c10f8f54ad9949dc62e76dacdcb 2023-03-24 16:54:44,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:44,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:44,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676944714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676944714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676944715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676944715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,808 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e504259c7c594b128fe2f2a7ab50f81a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e504259c7c594b128fe2f2a7ab50f81a 2023-03-24 16:54:44,809 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174 2023-03-24 16:54:44,812 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into e504259c7c594b128fe2f2a7ab50f81a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:44,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:44,812 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676884381; duration=0sec 2023-03-24 16:54:44,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:44,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:44,813 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into b3df1c4a8a654d4f8d9e2ae3c8d9e174(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:44,813 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:44,813 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676884381; duration=0sec 2023-03-24 16:54:44,813 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:44,814 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:44,814 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:44,814 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,814 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/61d718a294624603816cb2a1b6da4455, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=35.8 K 2023-03-24 16:54:44,814 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 61d718a294624603816cb2a1b6da4455, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679676881421 2023-03-24 16:54:44,814 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5e8cb50206bb4913a45a55e64c7a1f84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1679676881450 2023-03-24 16:54:44,815 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 2e1d25d7551c40988a7ea13499e2a90b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:44,825 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676944819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676944819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676944820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676944819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,879 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/1646207a900b40c3ba8ec4ba724f52ca 2023-03-24 16:54:44,909 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/0ac27dc8075f476e80507fb1ef8410b1 2023-03-24 16:54:44,914 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/f4b10c10f8f54ad9949dc62e76dacdcb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb 2023-03-24 16:54:44,919 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb, entries=150, sequenceid=522, filesize=11.9 K 2023-03-24 16:54:44,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/1646207a900b40c3ba8ec4ba724f52ca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca 2023-03-24 16:54:44,925 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca, entries=150, sequenceid=522, filesize=11.9 K 2023-03-24 16:54:44,925 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/0ac27dc8075f476e80507fb1ef8410b1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1 2023-03-24 16:54:44,929 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1, entries=150, sequenceid=522, filesize=11.9 K 2023-03-24 16:54:44,930 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 7d260a92475e999cfa4ef78eea03ae7b in 471ms, sequenceid=522, compaction requested=false 2023-03-24 16:54:44,930 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:44,931 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:44,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,933 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:44,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c5d2c23[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:44,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:44,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:44,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:44,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:44,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:44,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:44,942 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:44,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:44,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:44,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:44,942 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:44,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:44,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:44,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:44,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:44,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:44,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:44,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:44,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:44,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:44,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:44,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:44,950 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:44,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,950 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:44,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:44,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:44,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:44,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:45,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:45,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:45,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676945046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676885037, max timestamp=1679676885047Num uniques 859; 2023-03-24 16:54:45,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676885037, max timestamp=1679676885047Num uniques 859; 2023-03-24 16:54:45,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:45,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676885037, max timestamp=1679676885047Num uniques 859; 2023-03-24 16:54:45,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:45,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:45,050 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,053 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,056 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,056 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,056 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db7d2a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,066 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,068 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,068 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676945067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,070 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,070 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,070 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676945067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676945068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,079 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,079 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676945149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676945171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676945173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39206 deadline: 1679676945173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,179 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,183 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bbef24[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,191 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,191 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,191 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,198 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,202 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,202 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,250 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/e250af1f8f5d44a791a386f7b9e8a7dc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/e250af1f8f5d44a791a386f7b9e8a7dc 2023-03-24 16:54:45,255 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into e250af1f8f5d44a791a386f7b9e8a7dc(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:45,255 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:45,255 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676884381; duration=0sec 2023-03-24 16:54:45,255 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:45,302 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,306 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,308 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,310 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,310 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,310 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,310 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,312 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e21ce7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,314 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,314 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,315 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,317 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,319 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,319 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,326 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,326 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676945354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676945375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676945376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676945376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,427 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,430 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,434 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,434 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,434 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,434 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374b00ac[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,439 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/4006bb316c9144d9829a05b14a9da366 2023-03-24 16:54:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,447 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,447 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,447 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,447 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,447 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,451 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:45,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,455 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,455 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,455 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,455 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,555 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,559 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa41942[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,572 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,572 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,573 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,573 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,574 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,575 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,582 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,582 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676945607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,611 DEBUG [Thread-1699] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:45,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676945655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676945678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676945679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676945679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,682 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,685 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bb09fd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,701 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,701 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,703 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,703 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,703 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:45,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,707 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,707 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,808 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,811 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,819 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c2ec9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,821 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,825 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,825 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,838 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:45,839 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,839 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/d490278cdc9d4206ac632cd27d15e4c2 2023-03-24 16:54:45,939 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:45,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:45,944 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:45,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:45,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:45,945 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,945 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,946 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:45,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:45,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:45,947 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,947 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:45,947 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:45,948 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:45,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2d10a4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:45,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:45,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,952 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:45,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:45,952 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:45,952 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:45,954 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:45,956 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:45,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:45,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:45,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:45,956 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:45,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:45,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:45,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:45,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:45,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:45,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:45,963 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:45,963 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,063 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,066 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,070 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,070 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:46,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20896aaa[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,075 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,075 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,076 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,079 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,079 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,083 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:46,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,087 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,087 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:46,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676946161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676946184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676946184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676946185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,187 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,190 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,194 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,194 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,194 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:46,194 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,196 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f927ba7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,204 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,204 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,214 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:46,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,214 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/3d4be4a67e4d48efa6fed29c6fd19001 2023-03-24 16:54:46,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/4006bb316c9144d9829a05b14a9da366 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366 2023-03-24 16:54:46,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366, entries=250, sequenceid=536, filesize=16.7 K 2023-03-24 16:54:46,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/d490278cdc9d4206ac632cd27d15e4c2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2 2023-03-24 16:54:46,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2, entries=150, sequenceid=536, filesize=11.9 K 2023-03-24 16:54:46,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/3d4be4a67e4d48efa6fed29c6fd19001 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001 2023-03-24 16:54:46,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001, entries=150, sequenceid=536, filesize=11.9 K 2023-03-24 16:54:46,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 7d260a92475e999cfa4ef78eea03ae7b in 1255ms, sequenceid=536, compaction requested=true 2023-03-24 16:54:46,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:46,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:46,284 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:46,284 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:46,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:46,288 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41644 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:46,288 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:46,288 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,288 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e504259c7c594b128fe2f2a7ab50f81a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=40.7 K 2023-03-24 16:54:46,289 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:46,289 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:46,289 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,289 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e504259c7c594b128fe2f2a7ab50f81a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:46,289 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=35.9 K 2023-03-24 16:54:46,289 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f4b10c10f8f54ad9949dc62e76dacdcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1679676883592 2023-03-24 16:54:46,289 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b3df1c4a8a654d4f8d9e2ae3c8d9e174, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:46,290 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4006bb316c9144d9829a05b14a9da366, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:46,290 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1646207a900b40c3ba8ec4ba724f52ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1679676883592 2023-03-24 16:54:46,290 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d490278cdc9d4206ac632cd27d15e4c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:46,306 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:46,306 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:46,314 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,323 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,324 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,324 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,326 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,326 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,327 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:46,327 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:46,327 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/6b73f6c5231d4a10827db53456981c44 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/6b73f6c5231d4a10827db53456981c44 2023-03-24 16:54:46,332 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into 6b73f6c5231d4a10827db53456981c44(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:46,332 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,332 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676886284; duration=0sec 2023-03-24 16:54:46,333 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:46,333 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:46,334 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:46,334 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:46,334 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,334 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/e250af1f8f5d44a791a386f7b9e8a7dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=35.9 K 2023-03-24 16:54:46,334 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e250af1f8f5d44a791a386f7b9e8a7dc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679676883514 2023-03-24 16:54:46,335 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 0ac27dc8075f476e80507fb1ef8410b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1679676883592 2023-03-24 16:54:46,335 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3d4be4a67e4d48efa6fed29c6fd19001, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:46,340 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/14adba8a31334d7694efe1057cc69956 2023-03-24 16:54:46,345 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:46,362 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/ca848c8aa51e4e92821c042403805f4e 2023-03-24 16:54:46,386 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/ee5ad9c841f2422ca5d87c89b618009a 2023-03-24 16:54:46,390 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/14adba8a31334d7694efe1057cc69956 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956 2023-03-24 16:54:46,394 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956, entries=150, sequenceid=571, filesize=11.9 K 2023-03-24 16:54:46,396 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/ca848c8aa51e4e92821c042403805f4e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e 2023-03-24 16:54:46,403 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e, entries=150, sequenceid=571, filesize=11.9 K 2023-03-24 16:54:46,404 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/ee5ad9c841f2422ca5d87c89b618009a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a 2023-03-24 16:54:46,411 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a, entries=150, sequenceid=571, filesize=11.9 K 2023-03-24 16:54:46,412 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 85ms, sequenceid=571, compaction requested=false 2023-03-24 16:54:46,412 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,412 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0c7efa[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,418 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,424 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,424 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,431 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:46,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,432 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,432 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,532 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,535 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,537 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,539 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,539 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,539 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,539 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2926e30f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,543 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,543 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,544 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,544 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,548 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,549 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,555 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,555 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,656 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,659 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,660 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,663 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,663 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,663 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,663 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,664 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429110bb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,677 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,677 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,684 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,684 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,720 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/25be2a50a66d45a58ffa2d3b2330b09c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/25be2a50a66d45a58ffa2d3b2330b09c 2023-03-24 16:54:46,726 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 25be2a50a66d45a58ffa2d3b2330b09c(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:46,726 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,726 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676886284; duration=0sec 2023-03-24 16:54:46,726 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:46,759 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/f94ce4c095ee4377be90a35472c97bad as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f94ce4c095ee4377be90a35472c97bad 2023-03-24 16:54:46,764 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into f94ce4c095ee4377be90a35472c97bad(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:46,764 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,764 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676886284; duration=0sec 2023-03-24 16:54:46,764 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:46,784 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,788 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,794 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,801 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,801 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,801 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,801 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e82040d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,808 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,808 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,815 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,815 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,819 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,819 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,822 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,826 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,826 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:46,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,926 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:46,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:46,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:46,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:46,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:46,930 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:46,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:46,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:46,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:46,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:46,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:46,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:46,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:46,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:46,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:46,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56314289[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:46,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:46,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:46,941 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,941 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:46,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:46,945 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:46,945 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:46,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:46,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:46,946 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:46,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:46,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,948 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:46,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:46,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:46,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:46,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:46,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:46,953 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:46,953 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:46,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,053 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,064 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4891fddb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,075 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,075 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,076 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,077 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,080 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,080 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:47,086 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,087 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,087 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,087 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:47,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:47,180 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:47,180 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:47,180 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:47,180 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:47,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:47,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:47,187 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,191 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,195 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,199 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,199 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,200 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf28ba[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,203 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,203 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,203 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676887196, max timestamp=1679676887204Num uniques 859; 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676887196, max timestamp=1679676887204Num uniques 859; 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676887196, max timestamp=1679676887204Num uniques 859; 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:47,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,215 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,215 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676947218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676947219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676947220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,223 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,224 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,224 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,224 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676947223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,324 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676947324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676947324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,327 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,327 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,327 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,327 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676947324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676947327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,335 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,335 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,335 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,335 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce3e682[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,339 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,346 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,346 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:47,351 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,351 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,352 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,452 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,455 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,457 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,458 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,459 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,461 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509ad89d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,465 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,468 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,468 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,471 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,475 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,475 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,475 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:47,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,477 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,477 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,477 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676947526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676947526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676947529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676947531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,578 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,581 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,584 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,585 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,586 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17229870[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/5c747aea620a44f2a942ec39aa3bee91 2023-03-24 16:54:47,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,599 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,599 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,601 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,601 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,601 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,602 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,603 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,606 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,607 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,607 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,707 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,710 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,712 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,714 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,714 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,717 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,717 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,717 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd3e8d6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,720 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,720 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,739 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,739 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,741 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,741 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,741 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,742 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,744 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,747 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,747 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,748 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:47,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,748 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,748 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676947830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676947832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676947835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676947836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,849 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,852 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,852 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,852 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,852 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,852 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,852 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,854 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,856 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,856 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,856 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,856 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,857 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f97c438[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,860 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,860 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,860 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,862 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,865 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,865 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,868 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,872 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:47,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,873 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,873 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,873 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,872 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,873 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,873 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:47,973 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:47,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:47,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:47,976 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:47,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:47,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:47,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:47,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:47,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:47,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:47,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:47,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:47,982 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:47,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cfb567[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:47,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:47,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,988 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:47,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:47,989 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:47,992 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:47,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:47,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:47,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:47,992 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:47,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:47,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:47,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:47,997 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,998 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,000 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,000 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:48,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:48,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:48,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,001 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:48,001 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:48,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:48,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/d64db5e544e142ab9c198deb4745a895 2023-03-24 16:54:48,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/a0d1338b24444c2ab5bb76af93514b83 2023-03-24 16:54:48,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/5c747aea620a44f2a942ec39aa3bee91 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91 2023-03-24 16:54:48,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91, entries=150, sequenceid=589, filesize=11.9 K 2023-03-24 16:54:48,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/d64db5e544e142ab9c198deb4745a895 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895 2023-03-24 16:54:48,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895, entries=150, sequenceid=589, filesize=11.9 K 2023-03-24 16:54:48,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/a0d1338b24444c2ab5bb76af93514b83 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83 2023-03-24 16:54:48,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83, entries=150, sequenceid=589, filesize=11.9 K 2023-03-24 16:54:48,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7d260a92475e999cfa4ef78eea03ae7b in 849ms, sequenceid=589, compaction requested=true 2023-03-24 16:54:48,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:48,030 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:48,030 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:48,031 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:48,031 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:48,031 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:48,031 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:48,031 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:48,031 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/25be2a50a66d45a58ffa2d3b2330b09c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.0 K 2023-03-24 16:54:48,031 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:48,031 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/6b73f6c5231d4a10827db53456981c44, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.0 K 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 25be2a50a66d45a58ffa2d3b2330b09c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 6b73f6c5231d4a10827db53456981c44, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 14adba8a31334d7694efe1057cc69956, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679676885066 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ca848c8aa51e4e92821c042403805f4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679676885066 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5c747aea620a44f2a942ec39aa3bee91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:48,032 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d64db5e544e142ab9c198deb4745a895, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:48,037 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#242 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:48,038 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:48,050 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/6cc0996f159942de9f5e8e134659157d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6cc0996f159942de9f5e8e134659157d 2023-03-24 16:54:48,053 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 6cc0996f159942de9f5e8e134659157d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:48,053 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:48,053 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676888030; duration=0sec 2023-03-24 16:54:48,053 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:48,053 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:48,054 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:48,054 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:48,054 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:48,054 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f94ce4c095ee4377be90a35472c97bad, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.0 K 2023-03-24 16:54:48,055 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f94ce4c095ee4377be90a35472c97bad, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679676884708 2023-03-24 16:54:48,055 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ee5ad9c841f2422ca5d87c89b618009a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679676885066 2023-03-24 16:54:48,055 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a0d1338b24444c2ab5bb76af93514b83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:48,060 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:48,067 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/d7b4690be3d4482c84cb04396001f89c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d7b4690be3d4482c84cb04396001f89c 2023-03-24 16:54:48,070 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into d7b4690be3d4482c84cb04396001f89c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:48,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:48,070 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676888030; duration=0sec 2023-03-24 16:54:48,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:48,101 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:48,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:48,105 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:48,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:48,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:48,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:48,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:48,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:48,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:48,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:48,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:48,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:48,108 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:48,116 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3c204f0603b84046ad26f9506b08af7c 2023-03-24 16:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:48,335 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676948341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676948341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676948342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676948342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676948445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676948445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676948445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676948445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,451 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/15ebc511d0294422acd101181ab129c0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/15ebc511d0294422acd101181ab129c0 2023-03-24 16:54:48,456 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into 15ebc511d0294422acd101181ab129c0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:48,456 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:48,456 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676888030; duration=0sec 2023-03-24 16:54:48,456 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:48,526 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/9d719ed2fdd643ef987381053e84a928 2023-03-24 16:54:48,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676948647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676948647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676948647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676948648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,937 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4d5d5edc17924f48988c586f125fa464 2023-03-24 16:54:48,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3c204f0603b84046ad26f9506b08af7c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c 2023-03-24 16:54:48,945 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c, entries=150, sequenceid=627, filesize=11.9 K 2023-03-24 16:54:48,945 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/9d719ed2fdd643ef987381053e84a928 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928 2023-03-24 16:54:48,949 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928, entries=150, sequenceid=627, filesize=11.9 K 2023-03-24 16:54:48,950 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4d5d5edc17924f48988c586f125fa464 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464 2023-03-24 16:54:48,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676948950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676948950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676948952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676948952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,954 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464, entries=150, sequenceid=627, filesize=11.9 K 2023-03-24 16:54:48,954 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7d260a92475e999cfa4ef78eea03ae7b in 846ms, sequenceid=627, compaction requested=false 2023-03-24 16:54:48,954 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:48,955 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:48,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:48,958 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,958 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a2d5336[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:48,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:48,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:48,961 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,961 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:48,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:48,966 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:48,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:48,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:48,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:48,966 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:48,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:48,968 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:48,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:48,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:48,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:48,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:48,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:48,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:48,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:48,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:48,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:48,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:48,978 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:48,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:48,978 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:49,078 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:49,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:49,082 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:49,083 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,083 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:49,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:49,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:49,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:49,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:49,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:49,085 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:49,085 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:49,086 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:49,086 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:49,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676889462, max timestamp=1679676889467Num uniques 859; 2023-03-24 16:54:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676889462, max timestamp=1679676889467Num uniques 859; 2023-03-24 16:54:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676889462, max timestamp=1679676889467Num uniques 859; 2023-03-24 16:54:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:49,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676949479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676949479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676949480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676949481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,493 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/5e23e43ff73343d7a16fde8e4b4183b9 2023-03-24 16:54:49,503 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a927c6edd7ed4269b8daf93707808fd6 2023-03-24 16:54:49,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676949583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676949583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676949583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676949584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39246 deadline: 1679676949620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,622 DEBUG [Thread-1699] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:49,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676949787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676949787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676949788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:49,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676949788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,914 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/269ca6fffa6e4812b2e7093126c432c7 2023-03-24 16:54:49,919 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/5e23e43ff73343d7a16fde8e4b4183b9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9 2023-03-24 16:54:49,923 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9, entries=100, sequenceid=638, filesize=9.5 K 2023-03-24 16:54:49,924 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a927c6edd7ed4269b8daf93707808fd6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6 2023-03-24 16:54:49,933 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6, entries=100, sequenceid=638, filesize=9.5 K 2023-03-24 16:54:49,933 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/269ca6fffa6e4812b2e7093126c432c7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7 2023-03-24 16:54:49,937 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7, entries=100, sequenceid=638, filesize=9.5 K 2023-03-24 16:54:49,938 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7d260a92475e999cfa4ef78eea03ae7b in 853ms, sequenceid=638, compaction requested=true 2023-03-24 16:54:49,938 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:49,938 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:49,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:49,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94b81fd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:49,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:49,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:49,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:49,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:49,949 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:49,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:49,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:49,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:49,950 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:49,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:49,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:49,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,953 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:49,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:49,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:49,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:49,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:49,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:49,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:49,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:49,958 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:49,958 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:49,958 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:49,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:50,058 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:50,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:50,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:50,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:50,076 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:50,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:50,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:50,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:50,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:50,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:50,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:50,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:50,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:50,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:50,080 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:50,080 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:50,081 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:50,081 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:50,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:50,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676950099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676950100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676950101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676950101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676950204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676950207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676950207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676950207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676950408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676950410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676950410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676950411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,491 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/7fa349de6dc94d5f9711fb927f497033 2023-03-24 16:54:50,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676950711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676950712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676950713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676950714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:50,901 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a76a21a2500b40daa8f06d101d71cd27 2023-03-24 16:54:51,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676951215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:51,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676951216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676951217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676951217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,312 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8ff84b2c95bc4cbfa5397fc45ab24247 2023-03-24 16:54:51,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/7fa349de6dc94d5f9711fb927f497033 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033 2023-03-24 16:54:51,320 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033, entries=150, sequenceid=674, filesize=11.9 K 2023-03-24 16:54:51,320 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a76a21a2500b40daa8f06d101d71cd27 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27 2023-03-24 16:54:51,324 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27, entries=150, sequenceid=674, filesize=11.9 K 2023-03-24 16:54:51,325 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8ff84b2c95bc4cbfa5397fc45ab24247 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247 2023-03-24 16:54:51,328 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247, entries=150, sequenceid=674, filesize=11.9 K 2023-03-24 16:54:51,329 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7d260a92475e999cfa4ef78eea03ae7b in 1248ms, sequenceid=674, compaction requested=true 2023-03-24 16:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:51,332 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bc32e8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:51,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:51,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:51,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:51,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:51,338 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:51,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:51,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:51,339 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:51,339 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:51,340 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,340 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,340 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,340 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,342 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:51,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:51,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:51,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:51,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:51,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,348 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,348 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:51,449 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:51,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:51,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:51,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:51,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:51,452 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:51,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:51,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:51,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:51,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:51,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:51,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:51,456 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:51,864 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/9e26377d397843c6ac6378cf749e0a4b 2023-03-24 16:54:51,887 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a6dc19cb17294fbaafb52eebf4e5743a 2023-03-24 16:54:51,902 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/1fc7b7ba7a99433aa4f4779f17db27f9 2023-03-24 16:54:51,906 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/9e26377d397843c6ac6378cf749e0a4b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b 2023-03-24 16:54:51,910 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b, entries=100, sequenceid=684, filesize=9.5 K 2023-03-24 16:54:51,912 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a6dc19cb17294fbaafb52eebf4e5743a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a 2023-03-24 16:54:51,916 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a, entries=100, sequenceid=684, filesize=9.5 K 2023-03-24 16:54:51,917 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/1fc7b7ba7a99433aa4f4779f17db27f9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9 2023-03-24 16:54:51,921 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9, entries=100, sequenceid=684, filesize=9.5 K 2023-03-24 16:54:51,921 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 465ms, sequenceid=684, compaction requested=true 2023-03-24 16:54:51,921 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:51,922 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,923 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5cd215[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:51,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:51,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,927 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:51,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:51,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:51,931 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:51,931 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:51,933 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,933 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,933 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:51,933 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,935 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:51,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:51,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:51,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:51,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:51,937 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:51,937 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:51,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:52,038 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,041 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,043 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,045 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,045 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,045 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:52,045 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c789c4b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:52,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:52,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:52,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:52,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:52,059 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:52,059 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:52,059 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:52,059 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:52,059 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:52,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,067 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,068 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:52,068 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:52,168 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,172 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,176 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,176 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,176 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:52,176 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:52,178 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:52,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@969ba50[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:52,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:52,180 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:52,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:52,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:52,186 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:52,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:52,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:52,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:52,187 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:52,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:52,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,194 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,194 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:52,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:52,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:52,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/720b6ee2b24d4edc827df6931925356d 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676892241, max timestamp=1679676892249Num uniques 859; 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676892241, max timestamp=1679676892249Num uniques 859; 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676892241, max timestamp=1679676892249Num uniques 859; 2023-03-24 16:54:52,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:52,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676952267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676952267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676952268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676952271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,294 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,297 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,303 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,303 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,303 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:52,303 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:52,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,304 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20ebe2e7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:52,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:52,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,308 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:52,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:52,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:52,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:52,312 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:52,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:52,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:52,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:52,312 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:52,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,314 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:52,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,318 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:52,318 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,318 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:52,318 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676952372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676952372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676952374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676952375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,419 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,422 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,422 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,422 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,422 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,422 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,422 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:52,428 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:52,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0d4095[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:52,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:52,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,434 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:52,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:52,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:52,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:52,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:52,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:52,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:52,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:52,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,444 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:52,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:52,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,445 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:52,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,445 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:52,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:52,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,545 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,548 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:52,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:52,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:52,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120f0f5b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:52,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:52,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:52,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:52,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:52,562 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:52,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:52,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:52,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:52,562 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:52,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:52,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:52,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,566 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:52,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:52,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:52,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:52,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,569 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:52,569 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:52,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:52,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676952575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676952575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676952578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676952579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/fc382c41f0924a7488350bbbb5fea88f 2023-03-24 16:54:52,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/b4ba69be2a9044789b313aa64c3e4e77 2023-03-24 16:54:52,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/720b6ee2b24d4edc827df6931925356d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d 2023-03-24 16:54:52,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d, entries=200, sequenceid=697, filesize=14.3 K 2023-03-24 16:54:52,667 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/fc382c41f0924a7488350bbbb5fea88f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f 2023-03-24 16:54:52,669 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:52,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f, entries=150, sequenceid=697, filesize=11.9 K 2023-03-24 16:54:52,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/b4ba69be2a9044789b313aa64c3e4e77 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77 2023-03-24 16:54:52,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:52,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:52,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:52,673 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:52,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:52,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:52,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77, entries=150, sequenceid=697, filesize=11.9 K 2023-03-24 16:54:52,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:52,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:52,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,676 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7d260a92475e999cfa4ef78eea03ae7b in 448ms, sequenceid=697, compaction requested=true 2023-03-24 16:54:52,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:52,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:52,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:52,676 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:54:52,676 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:54:52,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:52,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:52,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:52,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:52,678 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71078 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:54:52,678 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:52,678 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:54:52,678 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,678 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:52,678 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6cc0996f159942de9f5e8e134659157d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=69.4 K 2023-03-24 16:54:52,678 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:52,678 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:52,679 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:52,679 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:54:52,679 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/15ebc511d0294422acd101181ab129c0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=67.0 K 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:52,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:52,679 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6cc0996f159942de9f5e8e134659157d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:52,680 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 15ebc511d0294422acd101181ab129c0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:52,680 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3c204f0603b84046ad26f9506b08af7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679676887214 2023-03-24 16:54:52,680 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9d719ed2fdd643ef987381053e84a928, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679676887214 2023-03-24 16:54:52,681 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5e23e43ff73343d7a16fde8e4b4183b9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1679676888340 2023-03-24 16:54:52,681 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7fa349de6dc94d5f9711fb927f497033, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679676889478 2023-03-24 16:54:52,681 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a927c6edd7ed4269b8daf93707808fd6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1679676888340 2023-03-24 16:54:52,682 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9e26377d397843c6ac6378cf749e0a4b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679676890099 2023-03-24 16:54:52,682 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 720b6ee2b24d4edc827df6931925356d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:52,683 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a76a21a2500b40daa8f06d101d71cd27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679676889478 2023-03-24 16:54:52,684 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a6dc19cb17294fbaafb52eebf4e5743a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679676890099 2023-03-24 16:54:52,685 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fc382c41f0924a7488350bbbb5fea88f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:52,705 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:52,709 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#262 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:52,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:52,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:52,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676952887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676952890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676952891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676952891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676952992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676952994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676952995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:52,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676952996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,096 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/7e4ac8fad98c481ea23f7029fdbc2602 2023-03-24 16:54:53,122 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/a603622540d547a6937a242600ea9418 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a603622540d547a6937a242600ea9418 2023-03-24 16:54:53,122 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/04131bc3a03243d2991981c80660e74b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/04131bc3a03243d2991981c80660e74b 2023-03-24 16:54:53,127 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into a603622540d547a6937a242600ea9418(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:53,127 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 04131bc3a03243d2991981c80660e74b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:53,127 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:53,127 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:53,128 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=10, startTime=1679676892676; duration=0sec 2023-03-24 16:54:53,128 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=10, startTime=1679676892676; duration=0sec 2023-03-24 16:54:53,128 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:53,128 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:53,128 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:54:53,130 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:54:53,130 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:53,130 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:53,130 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d7b4690be3d4482c84cb04396001f89c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=67.0 K 2023-03-24 16:54:53,130 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d7b4690be3d4482c84cb04396001f89c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1679676887176 2023-03-24 16:54:53,131 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4d5d5edc17924f48988c586f125fa464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679676887214 2023-03-24 16:54:53,131 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 269ca6fffa6e4812b2e7093126c432c7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1679676888340 2023-03-24 16:54:53,131 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8ff84b2c95bc4cbfa5397fc45ab24247, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679676889478 2023-03-24 16:54:53,131 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1fc7b7ba7a99433aa4f4779f17db27f9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679676890099 2023-03-24 16:54:53,132 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b4ba69be2a9044789b313aa64c3e4e77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:53,139 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:53,146 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/c409c4e7fcc14e81ad60f04033659d2b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/c409c4e7fcc14e81ad60f04033659d2b 2023-03-24 16:54:53,150 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into c409c4e7fcc14e81ad60f04033659d2b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:53,150 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:53,150 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=10, startTime=1679676892677; duration=0sec 2023-03-24 16:54:53,150 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:53,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676953198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676953198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676953200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676953200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676953499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676953500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676953504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676953504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,511 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/5a386cf4b2e7478b86f52bdb551aa3fa 2023-03-24 16:54:53,549 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/f9ebcecdcebf4287bbbe86cce80b64d4 2023-03-24 16:54:53,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/7e4ac8fad98c481ea23f7029fdbc2602 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602 2023-03-24 16:54:53,558 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602, entries=150, sequenceid=733, filesize=11.9 K 2023-03-24 16:54:53,559 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/5a386cf4b2e7478b86f52bdb551aa3fa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa 2023-03-24 16:54:53,563 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa, entries=150, sequenceid=733, filesize=11.9 K 2023-03-24 16:54:53,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/f9ebcecdcebf4287bbbe86cce80b64d4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4 2023-03-24 16:54:53,568 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4, entries=150, sequenceid=733, filesize=11.9 K 2023-03-24 16:54:53,569 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 7d260a92475e999cfa4ef78eea03ae7b in 889ms, sequenceid=733, compaction requested=false 2023-03-24 16:54:53,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:53,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:53,574 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f08c70[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:53,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:53,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,578 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:53,578 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:53,578 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:53,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:53,579 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:53,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:53,582 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:53,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:53,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:53,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:53,582 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:53,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:53,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:53,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:53,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:53,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:53,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:53,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:53,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,589 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:53,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:53,589 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:53,690 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:53,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:53,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:53,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:53,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:53,693 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:53,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:53,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:53,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:53,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:53,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:53,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:53,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:53,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:53,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:53,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:53,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:53,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:53,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:53,696 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:53,697 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:53,703 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/6ba032b24069491b9b47a8f02d140ffe 2023-03-24 16:54:54,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:54,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676894011, max timestamp=1679676894021Num uniques 859; 2023-03-24 16:54:54,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:54,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676894011, max timestamp=1679676894021Num uniques 859; 2023-03-24 16:54:54,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:54,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676894011, max timestamp=1679676894021Num uniques 859; 2023-03-24 16:54:54,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:54,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676954035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676954036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,114 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e61c127fb71a46df829eed0407001b40 2023-03-24 16:54:54,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676954138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676954139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676954344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676954344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,524 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/3b36aed559cf44a3a7e3c5fea9efe260 2023-03-24 16:54:54,531 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/6ba032b24069491b9b47a8f02d140ffe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe 2023-03-24 16:54:54,535 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe, entries=150, sequenceid=747, filesize=11.9 K 2023-03-24 16:54:54,536 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e61c127fb71a46df829eed0407001b40 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40 2023-03-24 16:54:54,539 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40, entries=150, sequenceid=747, filesize=11.9 K 2023-03-24 16:54:54,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/3b36aed559cf44a3a7e3c5fea9efe260 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260 2023-03-24 16:54:54,544 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260, entries=150, sequenceid=747, filesize=11.9 K 2023-03-24 16:54:54,545 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7d260a92475e999cfa4ef78eea03ae7b in 849ms, sequenceid=747, compaction requested=true 2023-03-24 16:54:54,545 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:54,545 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:54,545 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:54,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f3a093[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:54,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:54,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:54,556 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:54,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:54,556 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:54,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,566 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:54,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:54,566 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:54,566 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:54,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:54:54,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:54,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:54,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:54,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:54,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:54,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:54,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676954648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676954648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,666 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:54,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:54,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:54,670 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:54,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:54,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:54,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:54,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:54,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:54,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:54,676 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@491a54d0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:54,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:54,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:54,681 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:54,683 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:54,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:54,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:54,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:54,683 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:54,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:54,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:54,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,691 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,692 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:54,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:54,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,692 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,792 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:54,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:54,795 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:54,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:54,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:54,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,797 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,798 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:54,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:54,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:54,799 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,799 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:54,799 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:54,799 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2305ff7f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:54,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:54,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:54,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:54,811 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:54,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:54,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:54,811 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:54,812 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:54,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,818 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:54,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,818 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:54,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,819 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:54,919 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:54,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:54,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:54,923 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:54,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:54,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:54,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:54,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:54,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:54,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:54,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:54,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:54,927 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:54,929 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:54,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71446463[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:54,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:54,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:54,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:54,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:54,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:54,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:54,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,934 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:54,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:54,937 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:54,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:54,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:54,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:54,937 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:54,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:54,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:54,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:54,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:54,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:54,943 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:54,943 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:54,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:54,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:54,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676954954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:54,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676954954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e9a060b91da3400195fb5a07d17b5105 2023-03-24 16:54:55,044 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:55,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:55,047 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:55,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:55,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:55,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:55,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:55,051 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,051 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:55,051 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:55,051 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:55,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4587b5e0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:55,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:55,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:55,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:55,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:55,061 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:55,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:55,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:55,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:55,061 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:55,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,064 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:55,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:55,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,068 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:55,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,068 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:55,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676955153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676955154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,169 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:55,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:55,172 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:55,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:55,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:55,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:55,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:55,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:55,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:55,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:55,187 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea0c554[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:55,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:55,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:55,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,191 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:55,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:55,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:55,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:55,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:55,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:55,196 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:55,200 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,200 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,200 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:55,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:55,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:55,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,205 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:55,205 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:55,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,205 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676955257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676955258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,305 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:55,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:55,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:55,309 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:55,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:55,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:55,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,312 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,313 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:55,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:55,314 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,314 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:55,314 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:55,314 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:55,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525237f8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:55,317 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:55,317 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,319 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:55,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:55,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:55,323 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:55,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:55,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:55,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:55,323 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:55,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,326 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,327 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,329 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,329 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,329 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,329 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:55,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:55,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:55,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,330 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:55,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,330 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:55,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,430 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:55,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:55,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:55,436 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:55,436 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,436 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:55,436 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:55,437 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,437 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:55,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:55,442 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,442 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:55,442 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:55,442 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:55,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:55,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e726ad[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:55,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:55,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:55,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:55,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:55,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:55,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:55,457 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:55,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:55,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:55,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:55,457 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:55,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e127a645b5ff4ce197af8b780fd38940 2023-03-24 16:54:55,459 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,459 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:55,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,459 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:55,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:55,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:55,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:55,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,467 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:55,467 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:55,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:55,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:55,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/5187ad4f175c4b379900450cf5725891 2023-03-24 16:54:55,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e9a060b91da3400195fb5a07d17b5105 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105 2023-03-24 16:54:55,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105, entries=200, sequenceid=783, filesize=14.3 K 2023-03-24 16:54:55,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e127a645b5ff4ce197af8b780fd38940 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940 2023-03-24 16:54:55,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940, entries=150, sequenceid=783, filesize=11.9 K 2023-03-24 16:54:55,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/5187ad4f175c4b379900450cf5725891 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891 2023-03-24 16:54:55,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891, entries=150, sequenceid=783, filesize=11.9 K 2023-03-24 16:54:55,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7d260a92475e999cfa4ef78eea03ae7b in 898ms, sequenceid=783, compaction requested=true 2023-03-24 16:54:55,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:55,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:55,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:55,532 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:55,532 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:55,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:55,533 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51798 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:55,533 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:55,533 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:55,534 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:55,534 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,534 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/04131bc3a03243d2991981c80660e74b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=50.6 K 2023-03-24 16:54:55,534 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,534 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a603622540d547a6937a242600ea9418, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=48.2 K 2023-03-24 16:54:55,534 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 04131bc3a03243d2991981c80660e74b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:55,534 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a603622540d547a6937a242600ea9418, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:55,534 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7e4ac8fad98c481ea23f7029fdbc2602, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679676892265 2023-03-24 16:54:55,534 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5a386cf4b2e7478b86f52bdb551aa3fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679676892265 2023-03-24 16:54:55,535 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6ba032b24069491b9b47a8f02d140ffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1679676892879 2023-03-24 16:54:55,535 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e61c127fb71a46df829eed0407001b40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1679676892879 2023-03-24 16:54:55,535 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e9a060b91da3400195fb5a07d17b5105, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:55,535 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e127a645b5ff4ce197af8b780fd38940, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:55,545 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:55,545 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:55,559 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/af5eb2f9ece9428b854ba2489ab6d897 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/af5eb2f9ece9428b854ba2489ab6d897 2023-03-24 16:54:55,565 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into af5eb2f9ece9428b854ba2489ab6d897(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:55,566 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:55,566 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=12, startTime=1679676895532; duration=0sec 2023-03-24 16:54:55,566 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:55,566 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:55,567 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:55,567 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:55,567 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,568 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:55,568 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/c409c4e7fcc14e81ad60f04033659d2b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=48.2 K 2023-03-24 16:54:55,568 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c409c4e7fcc14e81ad60f04033659d2b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1679676892225 2023-03-24 16:54:55,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:55,568 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f9ebcecdcebf4287bbbe86cce80b64d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679676892265 2023-03-24 16:54:55,569 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3b36aed559cf44a3a7e3c5fea9efe260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1679676892879 2023-03-24 16:54:55,569 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5187ad4f175c4b379900450cf5725891, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:55,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:55,571 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:55,571 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,571 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:55,571 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:55,571 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,571 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,573 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:55,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:55,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:55,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:55,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:55,576 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:55,576 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:55,578 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#274 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:55,591 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/06a4a6319fb84bd2bf7fa10795b62cc6 2023-03-24 16:54:55,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:55,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:55,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676895777, max timestamp=1679676895790Num uniques 859; 2023-03-24 16:54:55,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676895777, max timestamp=1679676895790Num uniques 859; 2023-03-24 16:54:55,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:55,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676895777, max timestamp=1679676895790Num uniques 859; 2023-03-24 16:54:55,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:55,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676955812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676955814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676955915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676955917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:55,961 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/f9853f6269904a62a85a92f34b4dd1ff as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/f9853f6269904a62a85a92f34b4dd1ff 2023-03-24 16:54:55,965 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into f9853f6269904a62a85a92f34b4dd1ff(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:55,965 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:55,965 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=12, startTime=1679676895532; duration=0sec 2023-03-24 16:54:55,965 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:55,993 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/bf8106a1bca74fad81880b78475377ef as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/bf8106a1bca74fad81880b78475377ef 2023-03-24 16:54:55,997 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into bf8106a1bca74fad81880b78475377ef(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:55,997 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:55,997 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=12, startTime=1679676895532; duration=0sec 2023-03-24 16:54:55,997 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:56,023 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8d286e51aa214751bf0272aca8b0443b 2023-03-24 16:54:56,032 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8b5b90baf01d47ff9d312cc29a3ca279 2023-03-24 16:54:56,036 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/06a4a6319fb84bd2bf7fa10795b62cc6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6 2023-03-24 16:54:56,039 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6, entries=150, sequenceid=794, filesize=11.9 K 2023-03-24 16:54:56,040 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8d286e51aa214751bf0272aca8b0443b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b 2023-03-24 16:54:56,043 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b, entries=150, sequenceid=794, filesize=11.9 K 2023-03-24 16:54:56,044 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8b5b90baf01d47ff9d312cc29a3ca279 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279 2023-03-24 16:54:56,047 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279, entries=150, sequenceid=794, filesize=11.9 K 2023-03-24 16:54:56,047 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7d260a92475e999cfa4ef78eea03ae7b in 471ms, sequenceid=794, compaction requested=false 2023-03-24 16:54:56,048 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:56,048 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7837abe4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,056 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,056 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,069 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,070 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:56,121 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:56,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:56,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676956129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676956131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676956160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676956163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,170 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,173 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,175 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,176 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,179 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e54bd3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,182 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,184 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,187 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,188 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,196 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,196 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676956234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676956234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,296 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,303 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,308 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63cb55c0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,315 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,316 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,317 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,317 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,317 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,317 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,321 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,322 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,422 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,425 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,432 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa0d55a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676956436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676956436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,439 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,439 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,439 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,444 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,452 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,453 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,453 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,453 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e8339baf2bf64ab0992d66e1114b8de5 2023-03-24 16:54:56,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/31f1ecebdac847aaba426d148a256429 2023-03-24 16:54:56,553 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,562 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0d3d61[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,574 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,575 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,575 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,579 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,579 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,582 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,588 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,588 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,688 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,692 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,698 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15bc4680[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,702 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,702 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,705 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,705 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,710 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,711 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,711 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,711 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676956738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:56,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676956738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,811 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,814 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,816 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,816 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,817 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,818 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,818 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,818 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,818 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,820 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6064a2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,825 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,825 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,827 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,827 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,830 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,833 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,833 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,933 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:56,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:56,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:56,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:56,937 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:56,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:56,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:56,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:56,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:56,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:56,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:56,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:56,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:56,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:56,942 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:56,946 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70366e71[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:56,948 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:56,948 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:56,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:56,949 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:56,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:56,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:56,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4c9d1cd9fb614b1cb2f0522a2b24e71b 2023-03-24 16:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:56,954 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:56,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:56,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:56,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:56,955 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:56,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e8339baf2bf64ab0992d66e1114b8de5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5 2023-03-24 16:54:56,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:56,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5, entries=200, sequenceid=833, filesize=14.3 K 2023-03-24 16:54:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:56,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/31f1ecebdac847aaba426d148a256429 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429 2023-03-24 16:54:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:56,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:56,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:56,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:56,962 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:56,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,962 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:56,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:56,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:56,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429, entries=150, sequenceid=833, filesize=11.9 K 2023-03-24 16:54:56,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4c9d1cd9fb614b1cb2f0522a2b24e71b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b 2023-03-24 16:54:56,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b, entries=150, sequenceid=833, filesize=11.9 K 2023-03-24 16:54:56,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 7d260a92475e999cfa4ef78eea03ae7b in 848ms, sequenceid=833, compaction requested=true 2023-03-24 16:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:56,969 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:56,969 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:56,970 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:56,970 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:56,970 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:56,970 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,970 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:56,970 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/af5eb2f9ece9428b854ba2489ab6d897, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=38.8 K 2023-03-24 16:54:56,970 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,970 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/f9853f6269904a62a85a92f34b4dd1ff, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.4 K 2023-03-24 16:54:56,971 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting af5eb2f9ece9428b854ba2489ab6d897, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:56,971 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f9853f6269904a62a85a92f34b4dd1ff, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:56,971 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 06a4a6319fb84bd2bf7fa10795b62cc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1679676894645 2023-03-24 16:54:56,971 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8d286e51aa214751bf0272aca8b0443b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1679676894645 2023-03-24 16:54:56,972 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e8339baf2bf64ab0992d66e1114b8de5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:56,972 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 31f1ecebdac847aaba426d148a256429, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:56,978 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:56,979 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:56,993 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/2d89e3255be24256b0216393ba65cb94 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/2d89e3255be24256b0216393ba65cb94 2023-03-24 16:54:56,997 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into 2d89e3255be24256b0216393ba65cb94(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:56,998 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:56,998 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676896969; duration=0sec 2023-03-24 16:54:56,998 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:56,998 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:56,999 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:56,999 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:56,999 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:56,999 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/bf8106a1bca74fad81880b78475377ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.4 K 2023-03-24 16:54:56,999 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting bf8106a1bca74fad81880b78475377ef, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1679676894027 2023-03-24 16:54:56,999 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8b5b90baf01d47ff9d312cc29a3ca279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1679676894645 2023-03-24 16:54:57,000 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4c9d1cd9fb614b1cb2f0522a2b24e71b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:57,006 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:57,063 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,066 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,070 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,070 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,071 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:57,071 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,094 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3bb4830d554240948b0087b67d187961 2023-03-24 16:54:57,123 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8f616fedcf894250bc43aed513b5afca 2023-03-24 16:54:57,134 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8a6083505a764c70a5c06111d0933bab 2023-03-24 16:54:57,138 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/3bb4830d554240948b0087b67d187961 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961 2023-03-24 16:54:57,142 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 16:54:57,143 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8f616fedcf894250bc43aed513b5afca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca 2023-03-24 16:54:57,146 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 16:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/8a6083505a764c70a5c06111d0933bab as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab 2023-03-24 16:54:57,150 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 16:54:57,151 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 80ms, sequenceid=843, compaction requested=false 2023-03-24 16:54:57,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:57,151 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:57,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:57,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626b323[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:57,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:57,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,163 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:57,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:57,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:57,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:57,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:57,168 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:57,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:57,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:57,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:57,168 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:57,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,177 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:57,177 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:57,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:57,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:57,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:57,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:57,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/703028023e504816a707ac80c52d87a9 2023-03-24 16:54:57,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676897267, max timestamp=1679676897273Num uniques 859; 2023-03-24 16:54:57,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:57,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676897267, max timestamp=1679676897273Num uniques 859; 2023-03-24 16:54:57,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676897267, max timestamp=1679676897273Num uniques 859; 2023-03-24 16:54:57,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:57,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:54:57,277 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,280 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,282 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:57,284 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:57,286 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:57,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d96d21[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:57,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:57,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:57,291 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:57,293 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:57,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:57,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:57,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:57,293 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:57,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,301 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:57,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,301 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:57,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676957304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676957306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,388 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/204978a1e3dc4398a0750d72c9d8ad5e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/204978a1e3dc4398a0750d72c9d8ad5e 2023-03-24 16:54:57,392 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 204978a1e3dc4398a0750d72c9d8ad5e(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:57,393 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:57,393 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676896969; duration=0sec 2023-03-24 16:54:57,393 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:57,401 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,404 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,405 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,406 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,406 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:57,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:57,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676957407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:57,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb342cc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:57,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:57,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676957409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,416 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/3c43c820e52f4d5f844d5db349bde4da as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3c43c820e52f4d5f844d5db349bde4da 2023-03-24 16:54:57,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:57,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:57,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:57,418 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,421 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into 3c43c820e52f4d5f844d5db349bde4da(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:57,421 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:57,421 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676896969; duration=0sec 2023-03-24 16:54:57,421 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:57,421 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:57,421 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:57,421 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:57,421 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:57,421 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:57,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:57,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:57,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,428 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,428 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,528 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,531 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,531 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,532 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,533 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,535 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:57,535 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:57,535 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,535 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:57,537 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d35fa8d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:57,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:57,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:57,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:57,543 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:57,546 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:57,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:57,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:57,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:57,546 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:57,547 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,547 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,548 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,552 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,552 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:57,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:57,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676957610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676957614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,653 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:57,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:57,660 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:57,663 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:57,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e39eda3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:57,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:57,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:57,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:57,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/3e9b4ee44c04439f8d6a9763abbd41e7 2023-03-24 16:54:57,667 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:57,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:57,671 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:57,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:57,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:57,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:57,671 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:57,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:57,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:57,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/b5e54ce984ad4c4f8ae9f167c7ba441c 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:57,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:57,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,680 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,680 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:57,680 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:57,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:57,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/703028023e504816a707ac80c52d87a9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9 2023-03-24 16:54:57,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9, entries=150, sequenceid=853, filesize=11.9 K 2023-03-24 16:54:57,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/3e9b4ee44c04439f8d6a9763abbd41e7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7 2023-03-24 16:54:57,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7, entries=150, sequenceid=853, filesize=11.9 K 2023-03-24 16:54:57,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/b5e54ce984ad4c4f8ae9f167c7ba441c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c 2023-03-24 16:54:57,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c, entries=150, sequenceid=853, filesize=11.9 K 2023-03-24 16:54:57,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7d260a92475e999cfa4ef78eea03ae7b in 446ms, sequenceid=853, compaction requested=true 2023-03-24 16:54:57,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:57,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:57,694 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:57,694 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:57,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:57,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:57,697 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:57,698 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:57,698 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:57,698 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,698 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:57,698 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/204978a1e3dc4398a0750d72c9d8ad5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.5 K 2023-03-24 16:54:57,698 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,698 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/2d89e3255be24256b0216393ba65cb94, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.5 K 2023-03-24 16:54:57,698 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 204978a1e3dc4398a0750d72c9d8ad5e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:57,698 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2d89e3255be24256b0216393ba65cb94, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:57,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3bb4830d554240948b0087b67d187961, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679676896127 2023-03-24 16:54:57,699 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8f616fedcf894250bc43aed513b5afca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679676896127 2023-03-24 16:54:57,699 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3e9b4ee44c04439f8d6a9763abbd41e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:57,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 703028023e504816a707ac80c52d87a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:57,706 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#290 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:57,706 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#291 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:57,780 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:57,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:57,783 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:57,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:57,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:57,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,784 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,785 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:57,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:57,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:57,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:57,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:57,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:57,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:57,786 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:57,786 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:57,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:57,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:57,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:57,787 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:57,787 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:57,799 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=891 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/c6d2b2573584420fa60544a28f88d3bd 2023-03-24 16:54:57,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:57,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:57,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676957925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:57,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:57,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676957927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676958028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676958030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,124 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8a0fdd82a9fd4268ab0bd0d8de05706d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8a0fdd82a9fd4268ab0bd0d8de05706d 2023-03-24 16:54:58,126 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1 2023-03-24 16:54:58,138 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into e58f38f9f0fd4dbc9b4e7ae0949ccdb1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:58,138 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,138 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=13, startTime=1679676897694; duration=0sec 2023-03-24 16:54:58,138 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:58,138 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:54:58,138 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into 8a0fdd82a9fd4268ab0bd0d8de05706d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:58,138 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,138 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=13, startTime=1679676897694; duration=0sec 2023-03-24 16:54:58,138 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:58,139 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:54:58,139 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:58,139 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,139 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3c43c820e52f4d5f844d5db349bde4da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=36.5 K 2023-03-24 16:54:58,142 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3c43c820e52f4d5f844d5db349bde4da, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1679676895806 2023-03-24 16:54:58,143 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 8a6083505a764c70a5c06111d0933bab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679676896127 2023-03-24 16:54:58,143 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b5e54ce984ad4c4f8ae9f167c7ba441c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:58,148 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:58,160 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/a1583b306abe4762a43c8b1f0f579f33 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1583b306abe4762a43c8b1f0f579f33 2023-03-24 16:54:58,164 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into a1583b306abe4762a43c8b1f0f579f33(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:58,164 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,164 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=13, startTime=1679676897694; duration=0sec 2023-03-24 16:54:58,164 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:58,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39206 deadline: 1679676958170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,174 DEBUG [Thread-1701] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:58,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39288 deadline: 1679676958178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,180 DEBUG [Thread-1697] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:54:58,208 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=891 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e093c7e61d8c4b1693d9aa78e2afaa3a 2023-03-24 16:54:58,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676958232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676958233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676958535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:58,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676958535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,620 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=891 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/85df0d6ef87a49f98f1e6c1c17ec4f08 2023-03-24 16:54:58,625 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/c6d2b2573584420fa60544a28f88d3bd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd 2023-03-24 16:54:58,629 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd, entries=150, sequenceid=891, filesize=11.9 K 2023-03-24 16:54:58,630 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e093c7e61d8c4b1693d9aa78e2afaa3a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a 2023-03-24 16:54:58,634 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a, entries=150, sequenceid=891, filesize=11.9 K 2023-03-24 16:54:58,635 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/85df0d6ef87a49f98f1e6c1c17ec4f08 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08 2023-03-24 16:54:58,638 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08, entries=150, sequenceid=891, filesize=11.9 K 2023-03-24 16:54:58,639 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7d260a92475e999cfa4ef78eea03ae7b in 852ms, sequenceid=891, compaction requested=false 2023-03-24 16:54:58,639 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,639 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:58,641 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b32d20[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:58,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:58,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,644 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:58,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:58,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:58,649 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:58,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:58,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:58,649 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:58,649 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:58,654 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,654 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,654 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:58,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:58,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:58,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:58,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:58,659 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:58,659 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:58,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,659 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:58,759 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:58,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:58,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:58,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:58,763 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,765 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:58,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:58,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:58,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:58,767 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:58,774 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/924db478dd5042e5b516b9ddb68b3d76 2023-03-24 16:54:58,785 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/94b8c4277c4649039b6c0ff38d2d14fc 2023-03-24 16:54:58,795 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4ac1e3a9ddb64f51830674545d66b107 2023-03-24 16:54:58,799 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/924db478dd5042e5b516b9ddb68b3d76 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76 2023-03-24 16:54:58,802 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76, entries=150, sequenceid=904, filesize=11.9 K 2023-03-24 16:54:58,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/94b8c4277c4649039b6c0ff38d2d14fc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc 2023-03-24 16:54:58,807 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc, entries=150, sequenceid=904, filesize=11.9 K 2023-03-24 16:54:58,814 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/4ac1e3a9ddb64f51830674545d66b107 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107 2023-03-24 16:54:58,819 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107, entries=150, sequenceid=904, filesize=11.9 K 2023-03-24 16:54:58,820 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 53ms, sequenceid=904, compaction requested=true 2023-03-24 16:54:58,820 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,822 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:58,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5052b858[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:58,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:58,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:58,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:58,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:58,837 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:58,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:58,837 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:58,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,841 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:58,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:58,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,844 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:58,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:58,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:58,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:58,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,845 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:58,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,845 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:58,945 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:58,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:58,949 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:58,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:58,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:58,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:58,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:58,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:58,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:58,956 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:58,959 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6201d14d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:58,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:58,962 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,963 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,964 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:58,964 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:58,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:58,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:58,971 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:58,971 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:58,971 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:58,971 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:58,971 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:58,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:58,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,973 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:58,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:58,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:58,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:58,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:58,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:58,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:58,978 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:58,978 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:58,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:54:58,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:59,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:59,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:54:59,078 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:59,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:59,081 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,085 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,085 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:59,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:59,087 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,087 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:59,087 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:59,088 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:59,090 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3347c5b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:59,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:59,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:59,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:59,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:59,096 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:59,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:59,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:59,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:59,096 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:59,097 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676899087, max timestamp=1679676899096Num uniques 859; 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676899087, max timestamp=1679676899096Num uniques 859; 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676899087, max timestamp=1679676899096Num uniques 859; 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:54:59,098 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,098 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,099 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,100 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,106 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:59,107 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,107 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:59,107 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676959127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676959127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,207 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:59,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:59,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:59,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:59,211 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:59,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:59,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:59,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,212 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:59,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:59,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:59,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:59,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:59,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee38b30[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:59,218 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:59,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:59,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:59,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:59,224 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:59,224 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:59,224 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:59,224 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:59,224 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:59,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:59,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,230 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:59,230 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:59,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676959232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676959232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,330 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:59,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:59,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:59,333 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:59,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:59,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:59,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,335 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:59,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:59,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:59,337 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,337 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:59,337 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:59,337 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:59,340 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:59,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3662cc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:59,341 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:59,341 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:59,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,344 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:59,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:59,346 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:59,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:59,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:59,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:59,346 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:59,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,353 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:59,353 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,353 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676959434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676959435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,453 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:59,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:59,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:59,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:59,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:59,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:59,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:59,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/29b241bf808c4ec3afcafe971559abc5 2023-03-24 16:54:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:59,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:59,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:59,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:59,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:59,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:59,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:54:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:54:59,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:54:59,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d1595e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:54:59,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:54:59,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,465 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,465 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:54:59,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:54:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:54:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8d8c2fb739c34569babed20dc4ee8734 2023-03-24 16:54:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:54:59,472 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:54:59,472 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:54:59,472 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:54:59,472 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:54:59,472 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:54:59,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:54:59,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:54:59,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,476 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,482 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:54:59,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:54:59,482 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,482 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:54:59,482 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:54:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:54:59,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/d4d5ba55e7d1483985d2cce2bc8dae61 2023-03-24 16:54:59,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/29b241bf808c4ec3afcafe971559abc5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5 2023-03-24 16:54:59,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 16:54:59,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/8d8c2fb739c34569babed20dc4ee8734 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734 2023-03-24 16:54:59,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 16:54:59,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/d4d5ba55e7d1483985d2cce2bc8dae61 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61 2023-03-24 16:54:59,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 16:54:59,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7d260a92475e999cfa4ef78eea03ae7b in 468ms, sequenceid=916, compaction requested=true 2023-03-24 16:54:59,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:59,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:59,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:59,516 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:59,516 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:59,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:59,517 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:59,517 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/A is initiating minor compaction (all files) 2023-03-24 16:54:59,518 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/A in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,518 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:59,518 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=48.5 K 2023-03-24 16:54:59,518 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/B is initiating minor compaction (all files) 2023-03-24 16:54:59,518 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/B in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,518 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8a0fdd82a9fd4268ab0bd0d8de05706d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=48.5 K 2023-03-24 16:54:59,518 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e58f38f9f0fd4dbc9b4e7ae0949ccdb1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:59,518 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8a0fdd82a9fd4268ab0bd0d8de05706d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:59,518 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c6d2b2573584420fa60544a28f88d3bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=891, earliestPutTs=1679676897290 2023-03-24 16:54:59,519 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e093c7e61d8c4b1693d9aa78e2afaa3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=891, earliestPutTs=1679676897290 2023-03-24 16:54:59,519 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 924db478dd5042e5b516b9ddb68b3d76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1679676897923 2023-03-24 16:54:59,519 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 94b8c4277c4649039b6c0ff38d2d14fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1679676897923 2023-03-24 16:54:59,519 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 29b241bf808c4ec3afcafe971559abc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679676899044 2023-03-24 16:54:59,519 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8d8c2fb739c34569babed20dc4ee8734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679676899044 2023-03-24 16:54:59,527 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#B#compaction#303 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:59,527 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#A#compaction#302 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:54:59,582 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:54:59,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:54:59,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:54:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:54:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:54:59,593 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:54:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:54:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:54:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,598 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:54:59,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:54:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:54:59,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:54:59,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:54:59,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:54:59,602 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:54:59,609 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/337fdb1fd2ce477eab8469ebd269cb81 2023-03-24 16:54:59,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:54:59,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:54:59,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39246 deadline: 1679676959715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39274 deadline: 1679676959739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39222 deadline: 1679676959739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:54:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39246 deadline: 1679676959818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:54:59,943 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/1c58b453b07d44589b7d0d435137b72c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/1c58b453b07d44589b7d0d435137b72c 2023-03-24 16:54:59,944 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/311eab5126684f44b49e91f0676eeedc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/311eab5126684f44b49e91f0676eeedc 2023-03-24 16:54:59,947 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/A of 7d260a92475e999cfa4ef78eea03ae7b into 1c58b453b07d44589b7d0d435137b72c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:59,947 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:59,948 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/A, priority=12, startTime=1679676899516; duration=0sec 2023-03-24 16:54:59,948 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:54:59,948 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:54:59,948 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/B of 7d260a92475e999cfa4ef78eea03ae7b into 311eab5126684f44b49e91f0676eeedc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:54:59,948 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:54:59,948 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/B, priority=12, startTime=1679676899516; duration=0sec 2023-03-24 16:54:59,948 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:54:59,949 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:54:59,949 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 7d260a92475e999cfa4ef78eea03ae7b/C is initiating minor compaction (all files) 2023-03-24 16:54:59,949 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7d260a92475e999cfa4ef78eea03ae7b/C in TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:54:59,949 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1583b306abe4762a43c8b1f0f579f33, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp, totalSize=48.5 K 2023-03-24 16:54:59,950 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a1583b306abe4762a43c8b1f0f579f33, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=853, earliestPutTs=1679676897243 2023-03-24 16:54:59,951 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 85df0d6ef87a49f98f1e6c1c17ec4f08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=891, earliestPutTs=1679676897290 2023-03-24 16:54:59,951 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4ac1e3a9ddb64f51830674545d66b107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1679676897923 2023-03-24 16:54:59,951 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d4d5ba55e7d1483985d2cce2bc8dae61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679676899044 2023-03-24 16:54:59,958 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7d260a92475e999cfa4ef78eea03ae7b#C#compaction#306 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:00,019 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/630ea96b233249d79a73d974a1dd0a0f 2023-03-24 16:55:00,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:00,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39246 deadline: 1679676960021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,029 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/290abf7512034deb9d5fd4453a041a1f 2023-03-24 16:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/337fdb1fd2ce477eab8469ebd269cb81 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/337fdb1fd2ce477eab8469ebd269cb81 2023-03-24 16:55:00,037 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/337fdb1fd2ce477eab8469ebd269cb81, entries=150, sequenceid=952, filesize=11.9 K 2023-03-24 16:55:00,038 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/630ea96b233249d79a73d974a1dd0a0f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/630ea96b233249d79a73d974a1dd0a0f 2023-03-24 16:55:00,041 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/630ea96b233249d79a73d974a1dd0a0f, entries=150, sequenceid=952, filesize=11.9 K 2023-03-24 16:55:00,048 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/290abf7512034deb9d5fd4453a041a1f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/290abf7512034deb9d5fd4453a041a1f 2023-03-24 16:55:00,052 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/290abf7512034deb9d5fd4453a041a1f, entries=150, sequenceid=952, filesize=11.9 K 2023-03-24 16:55:00,052 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 7d260a92475e999cfa4ef78eea03ae7b in 450ms, sequenceid=952, compaction requested=false 2023-03-24 16:55:00,052 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:55:00,052 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:00,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:00,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a72f5b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:00,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:00,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:00,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:00,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:00,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:00,070 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:00,071 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:00,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:00,073 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:00,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:00,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:00,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:00,073 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:00,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:00,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,075 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:00,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:00,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:00,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:00,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:00,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:00,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:00,079 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:00,079 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:00,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:00,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:00,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:00,180 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:00,183 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:00,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:00,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:00,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:00,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:00,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:00,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:00,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:00,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:00,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:00,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:00,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:00,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. started... 2023-03-24 16:55:00,187 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:55:00,187 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:00,204 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=965 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/4a12e4db214149769011a7411ae951e2 2023-03-24 16:55:00,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:00,246 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. as already flushing 2023-03-24 16:55:00,264 DEBUG [Thread-1693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:62195 2023-03-24 16:55:00,264 DEBUG [Thread-1693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,265 DEBUG [Thread-1710] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:62195 2023-03-24 16:55:00,265 DEBUG [Thread-1710] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,265 DEBUG [Thread-1708] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:62195 2023-03-24 16:55:00,266 DEBUG [Thread-1708] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,266 DEBUG [Thread-1695] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:62195 2023-03-24 16:55:00,266 DEBUG [Thread-1695] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,267 DEBUG [Thread-1712] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:62195 2023-03-24 16:55:00,267 DEBUG [Thread-1712] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,268 DEBUG [Thread-1706] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:62195 2023-03-24 16:55:00,268 DEBUG [Thread-1706] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,269 DEBUG [Thread-1704] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:62195 2023-03-24 16:55:00,269 DEBUG [Thread-1704] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,325 DEBUG [Thread-1699] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:62195 2023-03-24 16:55:00,325 DEBUG [Thread-1699] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:00,367 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/ec9d15cd15e04f3aa1e0f157b6f23ff1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ec9d15cd15e04f3aa1e0f157b6f23ff1 2023-03-24 16:55:00,372 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7d260a92475e999cfa4ef78eea03ae7b/C of 7d260a92475e999cfa4ef78eea03ae7b into ec9d15cd15e04f3aa1e0f157b6f23ff1(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:00,372 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:55:00,372 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b., storeName=7d260a92475e999cfa4ef78eea03ae7b/C, priority=12, startTime=1679676899516; duration=0sec 2023-03-24 16:55:00,372 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:00,618 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=965 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/c47c050fcfd74b168819596aceb28961 2023-03-24 16:55:01,026 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=965 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/a1ee5c6742834eecad53ec230754c36b 2023-03-24 16:55:01,030 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/4a12e4db214149769011a7411ae951e2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4a12e4db214149769011a7411ae951e2 2023-03-24 16:55:01,032 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4a12e4db214149769011a7411ae951e2, entries=150, sequenceid=965, filesize=11.9 K 2023-03-24 16:55:01,033 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/c47c050fcfd74b168819596aceb28961 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/c47c050fcfd74b168819596aceb28961 2023-03-24 16:55:01,036 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/c47c050fcfd74b168819596aceb28961, entries=150, sequenceid=965, filesize=11.9 K 2023-03-24 16:55:01,037 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/a1ee5c6742834eecad53ec230754c36b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1ee5c6742834eecad53ec230754c36b 2023-03-24 16:55:01,040 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1ee5c6742834eecad53ec230754c36b, entries=150, sequenceid=965, filesize=11.9 K 2023-03-24 16:55:01,040 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=100.63 KB/103050 for 7d260a92475e999cfa4ef78eea03ae7b in 853ms, sequenceid=965, compaction requested=true 2023-03-24 16:55:01,041 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:55:01,041 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:01,042 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150f8e94[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:01,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:01,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:01,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:01,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:01,047 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:01,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:01,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:01,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:01,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:01,049 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:01,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:01,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:01,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:01,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:01,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:01,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:01,054 DEBUG [Thread-1703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,054 DEBUG [Thread-1703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:01,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:01,154 DEBUG [Thread-1703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:01,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:02,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:02,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:55:02,205 DEBUG [Thread-1701] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:62195 2023-03-24 16:55:02,205 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:55:02,205 DEBUG [Thread-1701] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:02,205 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:02,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:55:02,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:02,206 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:55:02,206 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:02,211 DEBUG [Thread-1697] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:62195 2023-03-24 16:55:02,211 DEBUG [Thread-1697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 231 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 229 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 168 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 105 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-03-24 16:55:02,211 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 5899 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 6061 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 5710 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 5886 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 6003 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:55:02,212 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:55:02,212 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112921f2 to 127.0.0.1:62195 2023-03-24 16:55:02,212 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:02,213 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:55:02,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:55:02,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:02,216 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676902216"}]},"ts":"1679676902216"} 2023-03-24 16:55:02,217 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:55:02,218 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:55:02,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, UNASSIGN}] 2023-03-24 16:55:02,220 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, UNASSIGN 2023-03-24 16:55:02,220 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=7d260a92475e999cfa4ef78eea03ae7b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:02,220 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676902220"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676902220"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676902220"}]},"ts":"1679676902220"} 2023-03-24 16:55:02,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:02,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:02,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:02,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7d260a92475e999cfa4ef78eea03ae7b, disabling compactions & flushes 2023-03-24 16:55:02,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:02,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:02,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/074ee99b10f54b909a02b42823b69760 2023-03-24 16:55:02,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:03,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e16a7ebe140b47efb513577d416a8afb 2023-03-24 16:55:03,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:03,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/92dd5d2fc9b74060ad0f4c29f11559ea 2023-03-24 16:55:03,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/074ee99b10f54b909a02b42823b69760 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/074ee99b10f54b909a02b42823b69760 2023-03-24 16:55:03,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/074ee99b10f54b909a02b42823b69760, entries=150, sequenceid=985, filesize=11.9 K 2023-03-24 16:55:03,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/e16a7ebe140b47efb513577d416a8afb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e16a7ebe140b47efb513577d416a8afb 2023-03-24 16:55:03,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e16a7ebe140b47efb513577d416a8afb, entries=150, sequenceid=985, filesize=11.9 K 2023-03-24 16:55:03,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/92dd5d2fc9b74060ad0f4c29f11559ea as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/92dd5d2fc9b74060ad0f4c29f11559ea 2023-03-24 16:55:03,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/92dd5d2fc9b74060ad0f4c29f11559ea, entries=150, sequenceid=985, filesize=11.9 K 2023-03-24 16:55:03,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=6.71 KB/6870 for 7d260a92475e999cfa4ef78eea03ae7b in 1239ms, sequenceid=985, compaction requested=true 2023-03-24 16:55:03,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:55:03,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:03,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:03,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:03,444 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. because compaction request was cancelled 2023-03-24 16:55:03,444 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. because compaction request was cancelled 2023-03-24 16:55:03,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. after waiting 0 ms 2023-03-24 16:55:03,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:03,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:03,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7d260a92475e999cfa4ef78eea03ae7b 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 16:55:03,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:03,444 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. because compaction request was cancelled 2023-03-24 16:55:03,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=A 2023-03-24 16:55:03,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:03,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=B 2023-03-24 16:55:03,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:03,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7d260a92475e999cfa4ef78eea03ae7b, store=C 2023-03-24 16:55:03,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:03,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=989 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/969837a3e63048088fdb35f99f44e80a 2023-03-24 16:55:04,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=989 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/47ac37fceac547dc9514dd19f91ac46b 2023-03-24 16:55:04,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:04,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=989 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/2cf14cd5a82f4d9ead936b69c82d8a7d 2023-03-24 16:55:04,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/A/969837a3e63048088fdb35f99f44e80a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/969837a3e63048088fdb35f99f44e80a 2023-03-24 16:55:04,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/969837a3e63048088fdb35f99f44e80a, entries=50, sequenceid=989, filesize=7.1 K 2023-03-24 16:55:04,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/B/47ac37fceac547dc9514dd19f91ac46b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/47ac37fceac547dc9514dd19f91ac46b 2023-03-24 16:55:04,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/47ac37fceac547dc9514dd19f91ac46b, entries=50, sequenceid=989, filesize=7.1 K 2023-03-24 16:55:04,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/.tmp/C/2cf14cd5a82f4d9ead936b69c82d8a7d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2cf14cd5a82f4d9ead936b69c82d8a7d 2023-03-24 16:55:04,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2cf14cd5a82f4d9ead936b69c82d8a7d, entries=50, sequenceid=989, filesize=7.1 K 2023-03-24 16:55:04,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for 7d260a92475e999cfa4ef78eea03ae7b in 1240ms, sequenceid=989, compaction requested=true 2023-03-24 16:55:04,704 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/325b758ff4d74db2b54d2389afacff7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e504259c7c594b128fe2f2a7ab50f81a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/25be2a50a66d45a58ffa2d3b2330b09c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6cc0996f159942de9f5e8e134659157d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/04131bc3a03243d2991981c80660e74b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/af5eb2f9ece9428b854ba2489ab6d897, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/204978a1e3dc4398a0750d72c9d8ad5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5] to archive 2023-03-24 16:55:04,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:04,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3fd5c413976a4274b1f460fc8c3fb327 2023-03-24 16:55:04,708 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/55f0a71f8ae2439ab4c9b1693cc9643a 2023-03-24 16:55:04,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e3154b84155b48e1b2629c72fe7de1fa 2023-03-24 16:55:04,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/325b758ff4d74db2b54d2389afacff7c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/325b758ff4d74db2b54d2389afacff7c 2023-03-24 16:55:04,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c7de9a2c2f1544b0a35eb92b77a7e901 2023-03-24 16:55:04,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e504259c7c594b128fe2f2a7ab50f81a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e504259c7c594b128fe2f2a7ab50f81a 2023-03-24 16:55:04,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/27554dfc2e60469a95ea7c40a6aecd3d 2023-03-24 16:55:04,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/f4b10c10f8f54ad9949dc62e76dacdcb 2023-03-24 16:55:04,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4006bb316c9144d9829a05b14a9da366 2023-03-24 16:55:04,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/25be2a50a66d45a58ffa2d3b2330b09c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/25be2a50a66d45a58ffa2d3b2330b09c 2023-03-24 16:55:04,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/14adba8a31334d7694efe1057cc69956 2023-03-24 16:55:04,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6cc0996f159942de9f5e8e134659157d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6cc0996f159942de9f5e8e134659157d 2023-03-24 16:55:04,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5c747aea620a44f2a942ec39aa3bee91 2023-03-24 16:55:04,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3c204f0603b84046ad26f9506b08af7c 2023-03-24 16:55:04,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/5e23e43ff73343d7a16fde8e4b4183b9 2023-03-24 16:55:04,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7fa349de6dc94d5f9711fb927f497033 2023-03-24 16:55:04,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/9e26377d397843c6ac6378cf749e0a4b 2023-03-24 16:55:04,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/720b6ee2b24d4edc827df6931925356d 2023-03-24 16:55:04,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/04131bc3a03243d2991981c80660e74b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/04131bc3a03243d2991981c80660e74b 2023-03-24 16:55:04,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/7e4ac8fad98c481ea23f7029fdbc2602 2023-03-24 16:55:04,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/6ba032b24069491b9b47a8f02d140ffe 2023-03-24 16:55:04,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e9a060b91da3400195fb5a07d17b5105 2023-03-24 16:55:04,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/af5eb2f9ece9428b854ba2489ab6d897 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/af5eb2f9ece9428b854ba2489ab6d897 2023-03-24 16:55:04,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/06a4a6319fb84bd2bf7fa10795b62cc6 2023-03-24 16:55:04,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e8339baf2bf64ab0992d66e1114b8de5 2023-03-24 16:55:04,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/204978a1e3dc4398a0750d72c9d8ad5e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/204978a1e3dc4398a0750d72c9d8ad5e 2023-03-24 16:55:04,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/3bb4830d554240948b0087b67d187961 2023-03-24 16:55:04,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/e58f38f9f0fd4dbc9b4e7ae0949ccdb1 2023-03-24 16:55:04,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/703028023e504816a707ac80c52d87a9 2023-03-24 16:55:04,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/c6d2b2573584420fa60544a28f88d3bd 2023-03-24 16:55:04,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/924db478dd5042e5b516b9ddb68b3d76 2023-03-24 16:55:04,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/29b241bf808c4ec3afcafe971559abc5 2023-03-24 16:55:04,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d78c507ddd0f4fcbbbe48c77b98ed1e2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/6b73f6c5231d4a10827db53456981c44, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/15ebc511d0294422acd101181ab129c0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a603622540d547a6937a242600ea9418, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/f9853f6269904a62a85a92f34b4dd1ff, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/2d89e3255be24256b0216393ba65cb94, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8a0fdd82a9fd4268ab0bd0d8de05706d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734] to archive 2023-03-24 16:55:04,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:04,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/18e253766a4840aa92f07a75d511c982 2023-03-24 16:55:04,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/779db1695e614e548e73a9d4496c47f0 2023-03-24 16:55:04,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d78c507ddd0f4fcbbbe48c77b98ed1e2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d78c507ddd0f4fcbbbe48c77b98ed1e2 2023-03-24 16:55:04,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/359d1cdea49b4cc8b23b6f0b0cd6850e 2023-03-24 16:55:04,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/264855925d9d4f7aa12c7c3228aeef98 2023-03-24 16:55:04,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/b3df1c4a8a654d4f8d9e2ae3c8d9e174 2023-03-24 16:55:04,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/708dfd3b552d409e88bbfecf6c4f1b1b 2023-03-24 16:55:04,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/1646207a900b40c3ba8ec4ba724f52ca 2023-03-24 16:55:04,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/6b73f6c5231d4a10827db53456981c44 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/6b73f6c5231d4a10827db53456981c44 2023-03-24 16:55:04,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d490278cdc9d4206ac632cd27d15e4c2 2023-03-24 16:55:04,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/ca848c8aa51e4e92821c042403805f4e 2023-03-24 16:55:04,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/15ebc511d0294422acd101181ab129c0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/15ebc511d0294422acd101181ab129c0 2023-03-24 16:55:04,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/d64db5e544e142ab9c198deb4745a895 2023-03-24 16:55:04,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/9d719ed2fdd643ef987381053e84a928 2023-03-24 16:55:04,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a927c6edd7ed4269b8daf93707808fd6 2023-03-24 16:55:04,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a76a21a2500b40daa8f06d101d71cd27 2023-03-24 16:55:04,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a6dc19cb17294fbaafb52eebf4e5743a 2023-03-24 16:55:04,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a603622540d547a6937a242600ea9418 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/a603622540d547a6937a242600ea9418 2023-03-24 16:55:04,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/fc382c41f0924a7488350bbbb5fea88f 2023-03-24 16:55:04,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/5a386cf4b2e7478b86f52bdb551aa3fa 2023-03-24 16:55:04,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e61c127fb71a46df829eed0407001b40 2023-03-24 16:55:04,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/f9853f6269904a62a85a92f34b4dd1ff to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/f9853f6269904a62a85a92f34b4dd1ff 2023-03-24 16:55:04,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e127a645b5ff4ce197af8b780fd38940 2023-03-24 16:55:04,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d286e51aa214751bf0272aca8b0443b 2023-03-24 16:55:04,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/2d89e3255be24256b0216393ba65cb94 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/2d89e3255be24256b0216393ba65cb94 2023-03-24 16:55:04,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/31f1ecebdac847aaba426d148a256429 2023-03-24 16:55:04,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8f616fedcf894250bc43aed513b5afca 2023-03-24 16:55:04,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8a0fdd82a9fd4268ab0bd0d8de05706d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8a0fdd82a9fd4268ab0bd0d8de05706d 2023-03-24 16:55:04,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/3e9b4ee44c04439f8d6a9763abbd41e7 2023-03-24 16:55:04,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e093c7e61d8c4b1693d9aa78e2afaa3a 2023-03-24 16:55:04,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/94b8c4277c4649039b6c0ff38d2d14fc 2023-03-24 16:55:04,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/8d8c2fb739c34569babed20dc4ee8734 2023-03-24 16:55:04,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/61d718a294624603816cb2a1b6da4455, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/e250af1f8f5d44a791a386f7b9e8a7dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f94ce4c095ee4377be90a35472c97bad, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d7b4690be3d4482c84cb04396001f89c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/c409c4e7fcc14e81ad60f04033659d2b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/bf8106a1bca74fad81880b78475377ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3c43c820e52f4d5f844d5db349bde4da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1583b306abe4762a43c8b1f0f579f33, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61] to archive 2023-03-24 16:55:04,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:04,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/7b984f38fe7f40a2a779c000128d9d68 2023-03-24 16:55:04,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ba9487e805ab4b62bf33965ff19eabee 2023-03-24 16:55:04,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/61d718a294624603816cb2a1b6da4455 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/61d718a294624603816cb2a1b6da4455 2023-03-24 16:55:04,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0436de402cc54c4491e5d1b18b756c22 2023-03-24 16:55:04,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5e8cb50206bb4913a45a55e64c7a1f84 2023-03-24 16:55:04,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/e250af1f8f5d44a791a386f7b9e8a7dc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/e250af1f8f5d44a791a386f7b9e8a7dc 2023-03-24 16:55:04,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2e1d25d7551c40988a7ea13499e2a90b 2023-03-24 16:55:04,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/0ac27dc8075f476e80507fb1ef8410b1 2023-03-24 16:55:04,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f94ce4c095ee4377be90a35472c97bad to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f94ce4c095ee4377be90a35472c97bad 2023-03-24 16:55:04,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3d4be4a67e4d48efa6fed29c6fd19001 2023-03-24 16:55:04,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ee5ad9c841f2422ca5d87c89b618009a 2023-03-24 16:55:04,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d7b4690be3d4482c84cb04396001f89c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d7b4690be3d4482c84cb04396001f89c 2023-03-24 16:55:04,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a0d1338b24444c2ab5bb76af93514b83 2023-03-24 16:55:04,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4d5d5edc17924f48988c586f125fa464 2023-03-24 16:55:04,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/269ca6fffa6e4812b2e7093126c432c7 2023-03-24 16:55:04,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8ff84b2c95bc4cbfa5397fc45ab24247 2023-03-24 16:55:04,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/1fc7b7ba7a99433aa4f4779f17db27f9 2023-03-24 16:55:04,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/c409c4e7fcc14e81ad60f04033659d2b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/c409c4e7fcc14e81ad60f04033659d2b 2023-03-24 16:55:04,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b4ba69be2a9044789b313aa64c3e4e77 2023-03-24 16:55:04,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/f9ebcecdcebf4287bbbe86cce80b64d4 2023-03-24 16:55:04,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3b36aed559cf44a3a7e3c5fea9efe260 2023-03-24 16:55:04,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/bf8106a1bca74fad81880b78475377ef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/bf8106a1bca74fad81880b78475377ef 2023-03-24 16:55:04,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/5187ad4f175c4b379900450cf5725891 2023-03-24 16:55:04,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8b5b90baf01d47ff9d312cc29a3ca279 2023-03-24 16:55:04,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3c43c820e52f4d5f844d5db349bde4da to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/3c43c820e52f4d5f844d5db349bde4da 2023-03-24 16:55:04,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4c9d1cd9fb614b1cb2f0522a2b24e71b 2023-03-24 16:55:04,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/8a6083505a764c70a5c06111d0933bab 2023-03-24 16:55:04,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1583b306abe4762a43c8b1f0f579f33 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1583b306abe4762a43c8b1f0f579f33 2023-03-24 16:55:04,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/b5e54ce984ad4c4f8ae9f167c7ba441c 2023-03-24 16:55:04,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/85df0d6ef87a49f98f1e6c1c17ec4f08 2023-03-24 16:55:04,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/4ac1e3a9ddb64f51830674545d66b107 2023-03-24 16:55:04,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/d4d5ba55e7d1483985d2cce2bc8dae61 2023-03-24 16:55:04,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/recovered.edits/992.seqid, newMaxSeqId=992, maxSeqId=1 2023-03-24 16:55:04,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b. 2023-03-24 16:55:04,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7d260a92475e999cfa4ef78eea03ae7b: 2023-03-24 16:55:04,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:04,798 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=7d260a92475e999cfa4ef78eea03ae7b, regionState=CLOSED 2023-03-24 16:55:04,798 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676904798"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676904798"}]},"ts":"1679676904798"} 2023-03-24 16:55:04,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-24 16:55:04,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 7d260a92475e999cfa4ef78eea03ae7b, server=jenkins-hbase4.apache.org,33315,1679676819242 in 2.5780 sec 2023-03-24 16:55:04,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-24 16:55:04,802 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7d260a92475e999cfa4ef78eea03ae7b, UNASSIGN in 2.5810 sec 2023-03-24 16:55:04,802 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676904802"}]},"ts":"1679676904802"} 2023-03-24 16:55:04,803 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:55:04,804 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:55:04,805 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5920 sec 2023-03-24 16:55:06,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 16:55:06,321 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-24 16:55:06,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:55:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,323 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 16:55:06,324 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,329 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:06,331 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/recovered.edits] 2023-03-24 16:55:06,333 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/074ee99b10f54b909a02b42823b69760 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/074ee99b10f54b909a02b42823b69760 2023-03-24 16:55:06,334 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/1c58b453b07d44589b7d0d435137b72c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/1c58b453b07d44589b7d0d435137b72c 2023-03-24 16:55:06,335 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/337fdb1fd2ce477eab8469ebd269cb81 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/337fdb1fd2ce477eab8469ebd269cb81 2023-03-24 16:55:06,336 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4a12e4db214149769011a7411ae951e2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/4a12e4db214149769011a7411ae951e2 2023-03-24 16:55:06,337 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/969837a3e63048088fdb35f99f44e80a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/A/969837a3e63048088fdb35f99f44e80a 2023-03-24 16:55:06,338 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/311eab5126684f44b49e91f0676eeedc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/311eab5126684f44b49e91f0676eeedc 2023-03-24 16:55:06,339 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/47ac37fceac547dc9514dd19f91ac46b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/47ac37fceac547dc9514dd19f91ac46b 2023-03-24 16:55:06,340 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/630ea96b233249d79a73d974a1dd0a0f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/630ea96b233249d79a73d974a1dd0a0f 2023-03-24 16:55:06,341 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/c47c050fcfd74b168819596aceb28961 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/c47c050fcfd74b168819596aceb28961 2023-03-24 16:55:06,342 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e16a7ebe140b47efb513577d416a8afb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/B/e16a7ebe140b47efb513577d416a8afb 2023-03-24 16:55:06,344 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/290abf7512034deb9d5fd4453a041a1f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/290abf7512034deb9d5fd4453a041a1f 2023-03-24 16:55:06,345 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2cf14cd5a82f4d9ead936b69c82d8a7d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/2cf14cd5a82f4d9ead936b69c82d8a7d 2023-03-24 16:55:06,346 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/92dd5d2fc9b74060ad0f4c29f11559ea to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/92dd5d2fc9b74060ad0f4c29f11559ea 2023-03-24 16:55:06,347 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1ee5c6742834eecad53ec230754c36b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/a1ee5c6742834eecad53ec230754c36b 2023-03-24 16:55:06,348 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ec9d15cd15e04f3aa1e0f157b6f23ff1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/C/ec9d15cd15e04f3aa1e0f157b6f23ff1 2023-03-24 16:55:06,350 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/recovered.edits/992.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b/recovered.edits/992.seqid 2023-03-24 16:55:06,350 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/7d260a92475e999cfa4ef78eea03ae7b 2023-03-24 16:55:06,350 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:55:06,352 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,353 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:55:06,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:55:06,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:55:06,356 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676906356"}]},"ts":"9223372036854775807"} 2023-03-24 16:55:06,357 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:55:06,357 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7d260a92475e999cfa4ef78eea03ae7b, NAME => 'TestAcidGuarantees,,1679676878096.7d260a92475e999cfa4ef78eea03ae7b.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:55:06,357 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:55:06,358 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676906357"}]},"ts":"9223372036854775807"} 2023-03-24 16:55:06,359 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:55:06,361 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-03-24 16:55:06,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 16:55:06,425 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-24 16:55:06,436 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 233), OpenFileDescriptor=573 (was 540) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 366), ProcessCount=169 (was 170), AvailableMemoryMB=5602 (was 5698) 2023-03-24 16:55:06,447 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=169, AvailableMemoryMB=5602 2023-03-24 16:55:06,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:55:06,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:55:06,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:06,451 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:55:06,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-24 16:55:06,452 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:55:06,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:06,453 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:06,453 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 empty. 2023-03-24 16:55:06,454 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:06,454 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:55:06,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:06,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:06,861 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:55:06,862 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0496a4cb656870861b4baf5a724e8ad6, NAME => 'TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:55:07,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0496a4cb656870861b4baf5a724e8ad6, disabling compactions & flushes 2023-03-24 16:55:07,269 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. after waiting 0 ms 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,269 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,269 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:07,271 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:55:07,272 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676907271"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676907271"}]},"ts":"1679676907271"} 2023-03-24 16:55:07,272 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:55:07,273 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:55:07,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676907273"}]},"ts":"1679676907273"} 2023-03-24 16:55:07,274 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:55:07,277 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, ASSIGN}] 2023-03-24 16:55:07,278 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, ASSIGN 2023-03-24 16:55:07,279 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:55:07,430 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:07,430 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676907430"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676907430"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676907430"}]},"ts":"1679676907430"} 2023-03-24 16:55:07,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:07,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:07,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0496a4cb656870861b4baf5a724e8ad6, NAME => 'TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:55:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:55:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,587 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,588 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A 2023-03-24 16:55:07,588 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A 2023-03-24 16:55:07,588 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:07,588 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName A 2023-03-24 16:55:07,589 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:07,589 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,590 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B 2023-03-24 16:55:07,590 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B 2023-03-24 16:55:07,590 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:07,590 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName B 2023-03-24 16:55:07,591 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:07,591 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,591 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C 2023-03-24 16:55:07,591 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C 2023-03-24 16:55:07,592 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:07,592 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName C 2023-03-24 16:55:07,592 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:07,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:55:07,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:07,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:55:07,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0496a4cb656870861b4baf5a724e8ad6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73308381, jitterRate=0.09238000214099884}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:55:07,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:07,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., pid=37, masterSystemTime=1679676907582 2023-03-24 16:55:07,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:07,600 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:07,600 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676907600"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676907600"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676907600"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676907600"}]},"ts":"1679676907600"} 2023-03-24 16:55:07,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-24 16:55:07,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 in 170 msec 2023-03-24 16:55:07,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-24 16:55:07,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, ASSIGN in 325 msec 2023-03-24 16:55:07,604 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:55:07,604 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676907604"}]},"ts":"1679676907604"} 2023-03-24 16:55:07,604 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:55:07,606 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:55:07,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-24 16:55:08,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 16:55:08,557 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-24 16:55:08,557 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:08,565 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:08,566 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:08,568 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52128, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:08,569 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:55:08,570 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:55:08,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:55:08,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:55:08,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:08,981 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:55:08,982 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:55:08,982 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:55:08,984 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 16:55:08,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, REOPEN/MOVE}] 2023-03-24 16:55:08,987 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, REOPEN/MOVE 2023-03-24 16:55:08,987 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:08,987 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676908987"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676908987"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676908987"}]},"ts":"1679676908987"} 2023-03-24 16:55:08,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:09,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0496a4cb656870861b4baf5a724e8ad6, disabling compactions & flushes 2023-03-24 16:55:09,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. after waiting 0 ms 2023-03-24 16:55:09,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 16:55:09,146 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:09,147 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 0496a4cb656870861b4baf5a724e8ad6 to self. 2023-03-24 16:55:09,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,148 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=CLOSED 2023-03-24 16:55:09,148 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676909148"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676909148"}]},"ts":"1679676909148"} 2023-03-24 16:55:09,150 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-24 16:55:09,150 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 in 161 msec 2023-03-24 16:55:09,150 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=true 2023-03-24 16:55:09,301 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,301 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676909301"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676909301"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676909301"}]},"ts":"1679676909301"} 2023-03-24 16:55:09,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:09,456 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0496a4cb656870861b4baf5a724e8ad6, NAME => 'TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:55:09,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:55:09,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,459 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,460 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A 2023-03-24 16:55:09,460 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A 2023-03-24 16:55:09,460 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:09,460 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName A 2023-03-24 16:55:09,462 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:09,462 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,462 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B 2023-03-24 16:55:09,463 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B 2023-03-24 16:55:09,463 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:09,463 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName B 2023-03-24 16:55:09,463 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:09,463 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,464 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C 2023-03-24 16:55:09,464 DEBUG [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C 2023-03-24 16:55:09,464 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:09,464 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0496a4cb656870861b4baf5a724e8ad6 columnFamilyName C 2023-03-24 16:55:09,465 INFO [StoreOpener-0496a4cb656870861b4baf5a724e8ad6-1] regionserver.HStore(310): Store=0496a4cb656870861b4baf5a724e8ad6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:09,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:55:09,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0496a4cb656870861b4baf5a724e8ad6; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61890417, jitterRate=-0.07776091992855072}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:55:09,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:09,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., pid=42, masterSystemTime=1679676909454 2023-03-24 16:55:09,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,472 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,472 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676909472"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676909472"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676909472"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676909472"}]},"ts":"1679676909472"} 2023-03-24 16:55:09,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-24 16:55:09,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 in 171 msec 2023-03-24 16:55:09,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-24 16:55:09,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, REOPEN/MOVE in 488 msec 2023-03-24 16:55:09,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-24 16:55:09,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-03-24 16:55:09,477 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-03-24 16:55:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-24 16:55:09,479 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,483 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,484 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,491 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,491 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,496 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,496 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,503 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,504 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,508 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,509 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,514 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,514 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,518 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,519 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,524 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,524 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,531 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,531 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:09,536 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:09,550 DEBUG [hconnection-0x45c2925c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,550 DEBUG [hconnection-0x6f1c59bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,550 DEBUG [hconnection-0x1db04254-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,550 DEBUG [hconnection-0x5cdd7655-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,550 DEBUG [hconnection-0x44bc40fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,551 DEBUG [hconnection-0x6c7083d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:09,552 DEBUG [hconnection-0x2090ce0d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,552 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52134, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,553 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52160, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,553 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52188, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,557 DEBUG [hconnection-0x4ddba098-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,557 DEBUG [hconnection-0x536a4c39-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,565 DEBUG [hconnection-0xbe63754-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:09,565 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,566 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,566 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52164, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,567 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52204, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,567 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,569 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52212, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:09,570 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:09,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:09,573 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:09,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:09,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:55:09,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:09,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:09,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:09,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:09,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:09,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:09,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:09,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:09,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:09,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd7befb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:09,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:09,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:09,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:09,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:09,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:09,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:09,596 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:09,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:09,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:09,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:09,596 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:09,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,598 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676969596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676909585, max timestamp=1679676909599Num uniques 859; 2023-03-24 16:55:09,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676909585, max timestamp=1679676909599Num uniques 859; 2023-03-24 16:55:09,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,602 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,602 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:09,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676909585, max timestamp=1679676909599Num uniques 859; 2023-03-24 16:55:09,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:55:09,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:55:09,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:09,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:09,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,604 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:09,604 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:09,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676969615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676969616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676969617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676969619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,704 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:09,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:09,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:09,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676969702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:09,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:09,710 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:09,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:09,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:09,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,712 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:09,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:09,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:09,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:09,720 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,720 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:09,720 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:09,720 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:09,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676969720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676969725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676969725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676969725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,728 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:09,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406101b6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:09,731 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:09,731 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,733 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,733 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:09,733 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:09,733 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:09,733 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,733 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,734 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:09,737 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:09,737 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:09,737 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:09,737 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:09,737 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:09,738 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,738 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,738 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:09,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:09,742 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,743 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:09,743 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,742 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,743 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,743 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,843 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:09,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:09,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:09,847 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:09,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:09,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:09,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,849 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,849 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:09,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:09,850 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,850 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:09,850 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:09,850 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:09,852 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e023316[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:09,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:09,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,854 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:09,855 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,855 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:09,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:09,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:09,859 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:09,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:09,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:09,859 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:09,859 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:09,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,864 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:09,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:09,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:09,868 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:09,868 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:09,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676969908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676969924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676969929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676969930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676969930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,968 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:09,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:09,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:09,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:09,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:09,976 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:09,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:09,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:09,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,978 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,983 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:09,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:09,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:09,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:09,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:09,988 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,988 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:09,988 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:09,988 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:09,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1790696c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:09,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:09,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,994 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:09,994 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:09,994 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:09,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:09,995 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,995 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:09,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,998 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f0a3fac636d24fc98762283c4b621b7f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f0a3fac636d24fc98762283c4b621b7f 2023-03-24 16:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:10,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:10,000 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:10,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:10,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:10,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:10,000 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:10,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,005 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,007 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:10,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:10,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,010 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:10,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,009 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,010 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:10,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,110 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:10,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:10,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:10,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:10,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:10,113 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:10,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:10,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:10,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,115 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:10,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:10,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:10,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:10,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:10,119 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:10,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47603d03[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:10,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:10,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:10,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:10,126 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:10,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:10,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:10,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:10,126 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:10,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,138 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,142 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:10,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,142 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:10,142 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,142 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676970211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676970228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676970236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676970238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,242 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:10,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:10,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:10,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676970242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:10,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:10,252 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:10,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:10,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:10,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:10,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:10,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:10,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:10,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:10,262 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,262 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:10,262 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:10,262 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:10,264 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,265 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,265 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc4e8e8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:10,267 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:10,267 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,267 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:10,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:10,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:10,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:10,270 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:10,273 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:10,273 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:10,273 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:10,273 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:10,273 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:10,274 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,275 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,275 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:10,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,289 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,289 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:10,324 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:55:10,390 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:10,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:10,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:10,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:10,393 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:10,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:10,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:10,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,395 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,395 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:10,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:10,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:10,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:10,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:10,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:10,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:10,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:10,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:10,405 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:10,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6031e085[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:10,408 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:10,408 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:10,418 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/a07d1d7d2c2b4f15880d41d9e4669188 2023-03-24 16:55:10,419 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:10,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:10,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:10,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:10,424 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:10,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:10,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:10,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:10,425 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:10,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:10,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:10,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,431 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:10,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:10,436 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:10,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,437 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:10,437 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:10,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:10,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:10,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:10,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:10,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/87b8ab541944408daeebc779d1062336 2023-03-24 16:55:10,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/75b138be735945c08f4a0d194b0754f9 2023-03-24 16:55:10,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/a07d1d7d2c2b4f15880d41d9e4669188 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188 2023-03-24 16:55:10,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188, entries=200, sequenceid=16, filesize=31.9 K 2023-03-24 16:55:10,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/87b8ab541944408daeebc779d1062336 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336 2023-03-24 16:55:10,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336, entries=150, sequenceid=16, filesize=11.6 K 2023-03-24 16:55:10,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/75b138be735945c08f4a0d194b0754f9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9 2023-03-24 16:55:10,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9, entries=150, sequenceid=16, filesize=11.6 K 2023-03-24 16:55:10,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 0496a4cb656870861b4baf5a724e8ad6 in 913ms, sequenceid=16, compaction requested=false 2023-03-24 16:55:10,491 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 16:55:10,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:10,537 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:10,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:10,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:10,547 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:10,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:10,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:10,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,549 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:10,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:10,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:10,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,549 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:10,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:10,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:10,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:10,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:10,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:10,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:10,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:10,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:10,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:10,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:10,552 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:10,564 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:10,564 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:10,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:10,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:10,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676970744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676970746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676970751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676970754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676970755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676970856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676970856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676970865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:10,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676970865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:10,983 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303246561890b4fc545b194793cc8a98fb4eb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246561890b4fc545b194793cc8a98fb4eb 2023-03-24 16:55:11,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676971058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676971060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676971071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676971074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676971366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676971370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676971377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676971378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,398 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/3dd1bf87bdbc43c997bcd3243b07abbe 2023-03-24 16:55:11,425 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/5ee274cbc6b34ab8b63fd450282c268e 2023-03-24 16:55:11,457 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/cd9d35d1b27b410d8517eed2fcd21c6e 2023-03-24 16:55:11,462 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/3dd1bf87bdbc43c997bcd3243b07abbe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe 2023-03-24 16:55:11,467 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe, entries=150, sequenceid=49, filesize=25.1 K 2023-03-24 16:55:11,468 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/5ee274cbc6b34ab8b63fd450282c268e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e 2023-03-24 16:55:11,472 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e, entries=150, sequenceid=49, filesize=11.6 K 2023-03-24 16:55:11,473 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/cd9d35d1b27b410d8517eed2fcd21c6e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e 2023-03-24 16:55:11,478 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e, entries=150, sequenceid=49, filesize=11.6 K 2023-03-24 16:55:11,479 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for 0496a4cb656870861b4baf5a724e8ad6 in 927ms, sequenceid=49, compaction requested=false 2023-03-24 16:55:11,481 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:11,481 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:11,484 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,484 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:11,485 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:11,496 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fe2ee6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:11,496 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:11,498 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:11,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:11,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:11,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:11,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:11,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:11,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:11,504 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:11,504 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,504 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:11,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:11,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:11,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,506 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:11,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:11,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:11,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:11,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:11,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:11,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:11,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:11,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:11,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:11,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,511 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:11,511 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:11,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:11,611 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:11,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:11,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:11,615 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:11,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:11,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:11,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:11,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:11,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:11,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:11,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:11,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:11,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:11,620 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:11,621 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:11,621 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:11,621 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:11,621 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:11,621 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:11,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:11,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:11,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676971849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676971869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676971877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676971885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676971889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:11,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:11,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676971952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,050 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ea70413382e944a7a25dffa1a2546df4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ea70413382e944a7a25dffa1a2546df4 2023-03-24 16:55:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676972157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676972463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,467 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/b1b970c2fb2143688a4a970775971cbd 2023-03-24 16:55:12,877 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/8859eabdaf334bb680cf5f388db7cdc8 2023-03-24 16:55:12,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676972879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,888 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/b7b748e4d6474f728f1a8a2590500966 2023-03-24 16:55:12,892 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/b1b970c2fb2143688a4a970775971cbd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd 2023-03-24 16:55:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676972887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,895 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd, entries=150, sequenceid=61, filesize=25.1 K 2023-03-24 16:55:12,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676972892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,896 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/8859eabdaf334bb680cf5f388db7cdc8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8 2023-03-24 16:55:12,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:12,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676972898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,904 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8, entries=150, sequenceid=61, filesize=11.6 K 2023-03-24 16:55:12,905 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/b7b748e4d6474f728f1a8a2590500966 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966 2023-03-24 16:55:12,910 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966, entries=150, sequenceid=61, filesize=11.6 K 2023-03-24 16:55:12,911 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0496a4cb656870861b4baf5a724e8ad6 in 1291ms, sequenceid=61, compaction requested=true 2023-03-24 16:55:12,911 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:12,911 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:12,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:12,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:12,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:12,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:12,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,915 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:12,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:12,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:12,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32309d5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:12,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:12,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:12,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:12,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:12,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:12,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:12,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:12,923 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:12,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:12,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:12,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:12,923 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:12,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:12,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:12,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:12,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:12,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:12,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:12,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:12,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:12,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:12,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:12,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:12,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:12,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:12,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:12,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:12,931 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:12,931 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:12,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:12,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:12,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:12,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:13,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:13,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676973009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,031 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,034 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,034 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,034 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,034 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,035 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,038 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,040 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,040 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,040 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,040 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1ba059[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,043 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,051 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,051 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,057 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,057 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,057 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,057 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:13,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676973118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,157 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,160 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,164 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75415e57[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,175 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,175 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,186 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,186 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,286 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,289 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,290 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,290 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,291 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,295 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70cdc707[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,303 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,303 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,310 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,311 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,311 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:13,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676973322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,387 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d6af58a57e3440d1a97342392514e086 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d6af58a57e3440d1a97342392514e086 2023-03-24 16:55:13,411 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,414 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,418 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,418 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,418 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,418 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46217624[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,423 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,429 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,429 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,429 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,433 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,433 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,433 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,434 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,439 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,439 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,439 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,439 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:13,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,439 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,439 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,539 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,543 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,543 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,543 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,543 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,548 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,548 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,548 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,548 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff0fd1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,556 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,559 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,559 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,561 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,562 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,563 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,566 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:13,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,567 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,567 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:13,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676973629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,667 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,670 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,672 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,674 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e334a5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,680 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,680 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,682 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,685 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,685 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,695 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,695 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,794 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/daff293f82a145189e7ca3698ce8878d 2023-03-24 16:55:13,795 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,808 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,810 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,810 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,810 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,810 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d85f61[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,821 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,821 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,833 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,833 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,833 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,933 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:13,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:13,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:13,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:13,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:13,936 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:13,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:13,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:13,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:13,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:13,943 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d058508[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:13,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:13,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,946 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:13,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:13,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:13,949 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:13,949 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:13,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:13,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:13,952 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:13,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:13,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:13,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,956 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:13,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:13,956 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:13,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:13,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,057 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:14,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:14,060 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:14,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:14,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:14,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:14,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:14,063 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,063 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:14,063 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:14,063 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:14,065 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,065 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,065 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54fd96fe[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:14,066 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:14,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:14,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:14,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:14,069 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:14,070 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:14,070 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:14,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:14,072 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:14,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:14,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:14,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:14,072 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:14,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:14,075 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,076 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:14,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:14,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:14,078 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:14,078 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:14,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:14,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:14,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676974135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,178 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:14,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:14,182 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:14,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:14,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:14,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:14,187 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30f85f8c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:14,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:14,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:14,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:14,190 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:14,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:14,193 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:14,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:14,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:14,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:14,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:14,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:14,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:14,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:14,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:14,200 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:14,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:14,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:14,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:14,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:14,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:14,204 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:14,204 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:14,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:14,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f4c8b25f8f1e43f0982fa86ff47da848 2023-03-24 16:55:14,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/de5d6a03ad75453dbe246c077ce63dd6 2023-03-24 16:55:14,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/daff293f82a145189e7ca3698ce8878d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d 2023-03-24 16:55:14,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d, entries=200, sequenceid=82, filesize=31.9 K 2023-03-24 16:55:14,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f4c8b25f8f1e43f0982fa86ff47da848 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848 2023-03-24 16:55:14,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 16:55:14,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/de5d6a03ad75453dbe246c077ce63dd6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6 2023-03-24 16:55:14,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 16:55:14,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 0496a4cb656870861b4baf5a724e8ad6 in 1270ms, sequenceid=82, compaction requested=true 2023-03-24 16:55:14,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:14,242 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:14,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:14,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:14,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:14,243 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:14,248 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 116864 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:14,248 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:14,248 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,248 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=114.1 K 2023-03-24 16:55:14,248 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:14,248 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:14,248 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,248 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a07d1d7d2c2b4f15880d41d9e4669188, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679676909566 2023-03-24 16:55:14,248 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=46.4 K 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3dd1bf87bdbc43c997bcd3243b07abbe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679676909613 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 87b8ab541944408daeebc779d1062336, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679676909575 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b1b970c2fb2143688a4a970775971cbd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679676910743 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5ee274cbc6b34ab8b63fd450282c268e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679676909613 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting daff293f82a145189e7ca3698ce8878d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:14,249 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8859eabdaf334bb680cf5f388db7cdc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679676910743 2023-03-24 16:55:14,250 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f4c8b25f8f1e43f0982fa86ff47da848, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:14,262 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#330 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:14,266 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#329 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:14,305 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:14,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:14,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:14,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:14,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:14,307 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:14,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,311 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:14,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:14,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:14,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:14,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:14,313 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:14,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:14,339 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032422c3dfa4ff204836a4b3dbe5333351a6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422c3dfa4ff204836a4b3dbe5333351a6 2023-03-24 16:55:14,675 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/3e6194d90f954f4cb93a2576bb7b4a9b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3e6194d90f954f4cb93a2576bb7b4a9b 2023-03-24 16:55:14,679 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 3e6194d90f954f4cb93a2576bb7b4a9b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:14,680 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:14,680 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=12, startTime=1679676914243; duration=0sec 2023-03-24 16:55:14,680 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:14,680 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:14,681 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:14,681 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:14,682 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:14,682 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=46.4 K 2023-03-24 16:55:14,682 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 75b138be735945c08f4a0d194b0754f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679676909575 2023-03-24 16:55:14,682 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cd9d35d1b27b410d8517eed2fcd21c6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679676909613 2023-03-24 16:55:14,683 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b7b748e4d6474f728f1a8a2590500966, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679676910743 2023-03-24 16:55:14,683 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting de5d6a03ad75453dbe246c077ce63dd6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:14,690 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:14,746 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=93, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/c77f24791c9246dba6fc06fc21d80af1 2023-03-24 16:55:14,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:14,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:14,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:14,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676974932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676914918, max timestamp=1679676914935Num uniques 859; 2023-03-24 16:55:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676914918, max timestamp=1679676914935Num uniques 859; 2023-03-24 16:55:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:14,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676914918, max timestamp=1679676914935Num uniques 859; 2023-03-24 16:55:14,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:14,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676974975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:14,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676974982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:14,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:14,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676974983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676975038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,085 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/dd3a746bb5b14a4ca916f7ff9aa1ee08 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/dd3a746bb5b14a4ca916f7ff9aa1ee08 2023-03-24 16:55:15,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676975084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,089 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into dd3a746bb5b14a4ca916f7ff9aa1ee08(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:15,089 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:15,089 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=12, startTime=1679676914242; duration=0sec 2023-03-24 16:55:15,089 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:15,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676975090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676975096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,101 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/bdeacac84eff4b949d412a7c6624a1bf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/bdeacac84eff4b949d412a7c6624a1bf 2023-03-24 16:55:15,105 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into bdeacac84eff4b949d412a7c6624a1bf(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:15,105 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:15,105 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=12, startTime=1679676914243; duration=0sec 2023-03-24 16:55:15,105 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:15,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676975144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,156 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/cbe51fa9d51a471884671514570e83f9 2023-03-24 16:55:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676975254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676975288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676975296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676975302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676975557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,565 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/e20e3065dc644fa1830e98de79b07034 2023-03-24 16:55:15,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/c77f24791c9246dba6fc06fc21d80af1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1 2023-03-24 16:55:15,573 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1, entries=150, sequenceid=93, filesize=25.1 K 2023-03-24 16:55:15,574 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/cbe51fa9d51a471884671514570e83f9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9 2023-03-24 16:55:15,577 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9, entries=150, sequenceid=93, filesize=11.6 K 2023-03-24 16:55:15,578 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/e20e3065dc644fa1830e98de79b07034 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034 2023-03-24 16:55:15,581 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034, entries=150, sequenceid=93, filesize=11.6 K 2023-03-24 16:55:15,581 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0496a4cb656870861b4baf5a724e8ad6 in 1268ms, sequenceid=93, compaction requested=false 2023-03-24 16:55:15,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:15,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:15,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:15,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:15,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:15,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,583 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,583 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f93da5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:15,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:15,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,586 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:15,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:15,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:15,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:15,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:15,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:15,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:15,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:15,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:15,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:15,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,594 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,596 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:15,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:15,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:15,597 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,597 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,597 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:15,597 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:15,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:15,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:55:15,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:15,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:15,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:15,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:15,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:15,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:15,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676975618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676975620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676975624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,697 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:15,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:15,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:15,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:15,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:15,700 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:15,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:15,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,702 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,702 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:15,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:15,704 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,704 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:15,704 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:15,704 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:15,710 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:15,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1355c737[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:15,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:15,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:15,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:15,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:15,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:15,716 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:15,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:15,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:15,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:15,717 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:15,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,722 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:15,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,723 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:15,723 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:15,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676975725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676975725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676975727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,823 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:15,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:15,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:15,829 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:15,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:15,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:15,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,830 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:15,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:15,832 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:15,832 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,832 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:15,833 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:15,833 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:15,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b27e9c5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:15,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:15,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:15,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:15,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:15,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:15,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:15,842 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:15,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:15,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:15,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:15,842 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:15,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:15,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:15,847 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,848 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:15,848 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:15,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676975928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676975929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:15,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676975931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,948 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:15,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:15,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:15,954 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:15,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:15,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:15,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:15,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:15,960 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,960 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:15,960 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:15,960 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:15,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:15,969 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:15,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5951d712[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:15,971 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:15,971 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,972 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:15,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:15,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:15,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:15,976 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:15,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:15,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:15,977 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:15,977 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:15,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,978 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:15,979 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:15,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,980 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:15,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:15,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:15,984 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:15,984 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:15,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:15,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:15,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:15,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:15,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,018 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b88ced0b53e342d0a31c397846566fe6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b88ced0b53e342d0a31c397846566fe6 2023-03-24 16:55:16,025 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/9ad3cf30f61746469a68d2becdce24db 2023-03-24 16:55:16,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676976062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,084 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,087 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,089 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,090 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,091 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,091 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,091 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,091 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21edd88[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,096 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,097 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,101 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,103 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,104 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,105 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,110 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,111 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:16,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,211 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,214 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,223 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fcfce83[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,230 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,232 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,232 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,234 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,234 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,234 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:16,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,239 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,239 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,240 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676976236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676976239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676976239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,339 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,349 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,351 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,360 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,360 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,360 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb2717e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,369 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,376 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,376 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:16,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,386 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,386 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/5d8ac146fa0045e38cadabada6699793 2023-03-24 16:55:16,486 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,489 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,489 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,490 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,491 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,491 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,492 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8cca5d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,497 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,498 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,507 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,507 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,507 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,607 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,610 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,610 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,611 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,613 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,613 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,613 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,613 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,615 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,615 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,615 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,615 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,617 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88aaad9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,618 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,618 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,619 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,620 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,620 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,621 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,623 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,624 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,626 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,626 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,626 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:16,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,630 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,630 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,730 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,734 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,741 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,742 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,742 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,744 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,744 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,744 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:16,744 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:16,747 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,748 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:16,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f0d69d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:16,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:16,749 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,751 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:16,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:16,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676976746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676976746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676976746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,754 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:16,754 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:16,755 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:16,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:16,757 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:16,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:16,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:16,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:16,757 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:16,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:16,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:16,762 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:16,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:16,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,774 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:16,774 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:16,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/ef5220093c114afbb5f87fbeb10f80cf 2023-03-24 16:55:16,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/9ad3cf30f61746469a68d2becdce24db as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db 2023-03-24 16:55:16,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db, entries=200, sequenceid=132, filesize=32.0 K 2023-03-24 16:55:16,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/5d8ac146fa0045e38cadabada6699793 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793 2023-03-24 16:55:16,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793, entries=150, sequenceid=132, filesize=11.7 K 2023-03-24 16:55:16,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/ef5220093c114afbb5f87fbeb10f80cf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf 2023-03-24 16:55:16,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf, entries=150, sequenceid=132, filesize=11.7 K 2023-03-24 16:55:16,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0496a4cb656870861b4baf5a724e8ad6 in 1268ms, sequenceid=132, compaction requested=true 2023-03-24 16:55:16,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:16,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:16,873 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:16,873 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:16,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:16,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:16,874 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:16,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:16,875 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:16,876 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:16,876 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,876 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35846 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:16,876 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/dd3a746bb5b14a4ca916f7ff9aa1ee08, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=82.4 K 2023-03-24 16:55:16,876 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:16,876 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,876 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3e6194d90f954f4cb93a2576bb7b4a9b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=35.0 K 2023-03-24 16:55:16,876 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting dd3a746bb5b14a4ca916f7ff9aa1ee08, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:16,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:16,876 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3e6194d90f954f4cb93a2576bb7b4a9b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:16,877 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c77f24791c9246dba6fc06fc21d80af1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679676912994 2023-03-24 16:55:16,877 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cbe51fa9d51a471884671514570e83f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679676912994 2023-03-24 16:55:16,877 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9ad3cf30f61746469a68d2becdce24db, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:16,877 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5d8ac146fa0045e38cadabada6699793, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:16,878 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:16,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:16,881 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:16,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:16,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:16,883 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:16,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:16,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:16,884 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:55:16,884 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:16,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:16,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:16,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:16,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:16,885 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:16,887 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#338 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:16,916 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324067a87b3399245ee81a3e0861e5a8c72 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324067a87b3399245ee81a3e0861e5a8c72 2023-03-24 16:55:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:17,090 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676917153, max timestamp=1679676917178Num uniques 859; 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676917153, max timestamp=1679676917178Num uniques 859; 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676917153, max timestamp=1679676917178Num uniques 859; 2023-03-24 16:55:17,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:17,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676977229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676977237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,309 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/9fdf4163f6944b999573618978129244 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/9fdf4163f6944b999573618978129244 2023-03-24 16:55:17,314 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 9fdf4163f6944b999573618978129244(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:17,314 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:17,314 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=13, startTime=1679676916873; duration=0sec 2023-03-24 16:55:17,314 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:17,315 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:17,315 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35846 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:17,316 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:17,316 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:17,316 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/bdeacac84eff4b949d412a7c6624a1bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=35.0 K 2023-03-24 16:55:17,316 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting bdeacac84eff4b949d412a7c6624a1bf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679676911832 2023-03-24 16:55:17,316 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e20e3065dc644fa1830e98de79b07034, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679676912994 2023-03-24 16:55:17,317 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ef5220093c114afbb5f87fbeb10f80cf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:17,319 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/d8f70a44fe6e4237b3446f38865c89ea as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d8f70a44fe6e4237b3446f38865c89ea 2023-03-24 16:55:17,321 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/7c20849f45bf48f6adb493914b4459fb 2023-03-24 16:55:17,323 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:17,324 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into d8f70a44fe6e4237b3446f38865c89ea(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:17,324 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:17,324 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=13, startTime=1679676916873; duration=0sec 2023-03-24 16:55:17,324 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:17,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676977340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676977346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,348 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/61e255a718d9491aa6c19b6f93fd0800 2023-03-24 16:55:17,357 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/d3a16620986641a0a30b2ff3b4646923 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d3a16620986641a0a30b2ff3b4646923 2023-03-24 16:55:17,367 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into d3a16620986641a0a30b2ff3b4646923(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:17,367 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:17,367 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=13, startTime=1679676916874; duration=0sec 2023-03-24 16:55:17,367 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:17,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676977547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676977547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676977756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676977763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676977764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,773 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/5008a5897a514edab37239debdaa7f22 2023-03-24 16:55:17,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/7c20849f45bf48f6adb493914b4459fb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb 2023-03-24 16:55:17,781 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb, entries=150, sequenceid=142, filesize=25.3 K 2023-03-24 16:55:17,782 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/61e255a718d9491aa6c19b6f93fd0800 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800 2023-03-24 16:55:17,786 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800, entries=150, sequenceid=142, filesize=11.8 K 2023-03-24 16:55:17,786 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/5008a5897a514edab37239debdaa7f22 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22 2023-03-24 16:55:17,789 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22, entries=150, sequenceid=142, filesize=11.8 K 2023-03-24 16:55:17,790 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0496a4cb656870861b4baf5a724e8ad6 in 906ms, sequenceid=142, compaction requested=false 2023-03-24 16:55:17,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:17,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:17,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:17,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,796 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:17,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f1675c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:17,798 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:17,798 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,799 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:17,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:17,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:17,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:17,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:17,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:17,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:17,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:17,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:17,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:17,804 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,804 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,804 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:17,805 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,810 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:17,809 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:17,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:17,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:17,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:17,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,810 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:17,810 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:17,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:17,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:17,857 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:17,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676977880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676977881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,910 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:17,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:17,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:17,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:17,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:17,914 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:17,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:17,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:17,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:17,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,916 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:17,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:17,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:17,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:17,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:17,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:17,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:17,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:17,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:17,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:17,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:17,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:17,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b35951d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:17,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:17,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:17,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:17,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:17,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:17,930 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:17,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:17,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:17,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:17,930 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:17,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:17,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:17,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:17,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,934 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,935 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:17,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:17,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:17,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:17,936 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:17,936 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:17,936 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:17,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:17,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:17,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676977986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676977991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,036 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,039 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,041 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,042 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,043 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,043 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,043 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,043 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363426c0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,048 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,048 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,050 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,050 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,052 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,052 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,053 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,055 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,063 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,063 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,063 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,063 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,163 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,167 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,170 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,170 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,170 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,170 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79bf4aa5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,179 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,181 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,181 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,187 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,187 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,187 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,187 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,188 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,187 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,188 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:18,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676978192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676978197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,283 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248fd16624024146f1abe0e9a88e48b7f7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fd16624024146f1abe0e9a88e48b7f7 2023-03-24 16:55:18,288 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,290 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,290 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,291 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,293 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,294 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,295 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,297 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e0949[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,301 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,302 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,305 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,305 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,308 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,311 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,313 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:18,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,314 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,314 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:18,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,423 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,426 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,432 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d4f958[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,434 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,436 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,437 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,447 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,448 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,448 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,448 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,448 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676978498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:18,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52188 deadline: 1679676978503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,548 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,551 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,555 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,555 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,555 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,555 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,557 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53c1d524[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,572 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,572 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,672 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,675 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:18,679 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,681 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:18,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a8579b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:18,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:18,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,684 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:18,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:18,689 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:18,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:18,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:18,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:18,689 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:18,689 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/447d2154d20541c582e76226371d977d 2023-03-24 16:55:18,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:18,691 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:18,696 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,697 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:18,697 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:18,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:18,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:18,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/b00674f7d13f4219a96aae6ddeea604d 2023-03-24 16:55:18,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/c8f952977ff2467e9e11eea89ec2bd05 2023-03-24 16:55:18,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/447d2154d20541c582e76226371d977d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d 2023-03-24 16:55:18,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d, entries=250, sequenceid=182, filesize=38.9 K 2023-03-24 16:55:18,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/b00674f7d13f4219a96aae6ddeea604d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d 2023-03-24 16:55:18,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 16:55:18,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/c8f952977ff2467e9e11eea89ec2bd05 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05 2023-03-24 16:55:18,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 16:55:18,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 0496a4cb656870861b4baf5a724e8ad6 in 871ms, sequenceid=182, compaction requested=true 2023-03-24 16:55:18,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:18,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:18,728 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:18,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:18,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:18,728 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:18,729 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91812 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:18,729 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:18,729 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36248 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:18,729 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,729 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:18,729 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d8f70a44fe6e4237b3446f38865c89ea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=89.7 K 2023-03-24 16:55:18,729 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,729 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/9fdf4163f6944b999573618978129244, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=35.4 K 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d8f70a44fe6e4237b3446f38865c89ea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9fdf4163f6944b999573618978129244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7c20849f45bf48f6adb493914b4459fb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679676915608 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 61e255a718d9491aa6c19b6f93fd0800, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679676915608 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 447d2154d20541c582e76226371d977d, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:18,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b00674f7d13f4219a96aae6ddeea604d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:18,737 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:18,745 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#347 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:18,797 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:18,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:18,800 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:18,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:18,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:18,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:18,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:18,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:18,802 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:18,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:18,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:18,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:18,803 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:18,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:19,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:19,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676919039, max timestamp=1679676919051Num uniques 859; 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676919039, max timestamp=1679676919051Num uniques 859; 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676919039, max timestamp=1679676919051Num uniques 859; 2023-03-24 16:55:19,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:19,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676919079, max timestamp=1679676919097Num uniques 1018; 2023-03-24 16:55:19,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:55:19,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676919079, max timestamp=1679676919097Num uniques 1018; 2023-03-24 16:55:19,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676919079, max timestamp=1679676919097Num uniques 1018; 2023-03-24 16:55:19,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:55:19,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676919129, max timestamp=1679676919145Num uniques 1077; 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676919129, max timestamp=1679676919145Num uniques 1077; 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676919129, max timestamp=1679676919145Num uniques 1077; 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:55:19,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:55:19,163 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/ecd6d7805f1c4a2aaee42abca5830a57 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ecd6d7805f1c4a2aaee42abca5830a57 2023-03-24 16:55:19,167 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into ecd6d7805f1c4a2aaee42abca5830a57(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:19,167 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:19,167 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=13, startTime=1679676918728; duration=0sec 2023-03-24 16:55:19,168 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:19,168 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:19,168 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36248 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:19,168 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:19,168 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:19,169 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d3a16620986641a0a30b2ff3b4646923, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=35.4 K 2023-03-24 16:55:19,169 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d3a16620986641a0a30b2ff3b4646923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1679676914963 2023-03-24 16:55:19,169 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5008a5897a514edab37239debdaa7f22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679676915608 2023-03-24 16:55:19,169 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting c8f952977ff2467e9e11eea89ec2bd05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:19,175 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#350 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:19,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676979215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,220 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324291a42d9212d45eda14af97ab37326b1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324291a42d9212d45eda14af97ab37326b1 2023-03-24 16:55:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676979216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,225 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/92f4129ee9894291a0374eebb5aaf53e 2023-03-24 16:55:19,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676979320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676979323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676979524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676979529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,584 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/a8f777372e614fa5bfb1b2b8e269a8fb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/a8f777372e614fa5bfb1b2b8e269a8fb 2023-03-24 16:55:19,588 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into a8f777372e614fa5bfb1b2b8e269a8fb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:19,588 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:19,588 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=13, startTime=1679676918728; duration=0sec 2023-03-24 16:55:19,589 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:19,592 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/d7fc907a3757429489887111983f9f8d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d7fc907a3757429489887111983f9f8d 2023-03-24 16:55:19,596 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into d7fc907a3757429489887111983f9f8d(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:19,596 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:19,596 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=13, startTime=1679676918728; duration=0sec 2023-03-24 16:55:19,596 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:19,633 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/3f47d84dd12143e8a91deb67085adb97 2023-03-24 16:55:19,642 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/2c97329015c046a8a0db6733d5e59d24 2023-03-24 16:55:19,645 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/92f4129ee9894291a0374eebb5aaf53e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e 2023-03-24 16:55:19,648 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e, entries=150, sequenceid=190, filesize=25.3 K 2023-03-24 16:55:19,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/3f47d84dd12143e8a91deb67085adb97 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97 2023-03-24 16:55:19,652 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97, entries=150, sequenceid=190, filesize=11.8 K 2023-03-24 16:55:19,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/2c97329015c046a8a0db6733d5e59d24 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24 2023-03-24 16:55:19,655 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24, entries=150, sequenceid=190, filesize=11.8 K 2023-03-24 16:55:19,656 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 0496a4cb656870861b4baf5a724e8ad6 in 853ms, sequenceid=190, compaction requested=false 2023-03-24 16:55:19,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:19,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:19,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@239fc886[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:19,660 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:19,660 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:19,662 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,662 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:19,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:19,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:19,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:19,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:19,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:19,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:19,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:19,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:19,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:19,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:19,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:19,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:19,669 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:19,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:19,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:19,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:19,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:19,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:19,670 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:19,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,670 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:19,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:19,770 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:19,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:19,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:19,773 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:19,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:19,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:19,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:19,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:19,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:19,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:19,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:19,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:19,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:19,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:19,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:19,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:19,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:19,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:19,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:19,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:19,776 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:19,776 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:19,776 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-03-24 16:55:19,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:19,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:19,777 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 16:55:19,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676979784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676979786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676979786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676979830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676979832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676979887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676979892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:19,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676979892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676980094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676980097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676980097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,191 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324260d79876a944c35b1880345b1f7d3c8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324260d79876a944c35b1880345b1f7d3c8 2023-03-24 16:55:20,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676980338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676980339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676980400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676980401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:20,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676980401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,596 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=265, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/a4d01872d7e241fea0d28470af8af341 2023-03-24 16:55:20,606 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/07ea8e09e25f4e59a98eafa32ee94afa 2023-03-24 16:55:20,618 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/5fd65494ee964d4db6841ba0a62efd52 2023-03-24 16:55:20,622 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/a4d01872d7e241fea0d28470af8af341 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341 2023-03-24 16:55:20,625 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341, entries=150, sequenceid=265, filesize=25.4 K 2023-03-24 16:55:20,625 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/07ea8e09e25f4e59a98eafa32ee94afa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa 2023-03-24 16:55:20,634 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa, entries=150, sequenceid=265, filesize=11.9 K 2023-03-24 16:55:20,637 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/5fd65494ee964d4db6841ba0a62efd52 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52 2023-03-24 16:55:20,645 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52, entries=150, sequenceid=265, filesize=11.9 K 2023-03-24 16:55:20,645 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 0496a4cb656870861b4baf5a724e8ad6 in 869ms, sequenceid=265, compaction requested=true 2023-03-24 16:55:20,645 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:20,645 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:20,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:20,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:20,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4062eb94[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:20,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:20,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:20,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:20,668 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:20,671 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:20,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:20,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:20,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:20,671 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:20,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:20,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:20,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:20,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:20,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:20,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:20,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:20,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:20,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,679 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:20,679 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:20,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:20,779 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:20,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:20,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:20,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:20,783 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:20,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:20,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:20,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:20,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,783 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:20,785 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:20,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:20,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:20,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:20,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:20,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:20,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:20,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:20,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:20,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:20,803 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:20,804 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:20,839 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ad295cae74a947a2a5ec897e45f1c50b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ad295cae74a947a2a5ec897e45f1c50b 2023-03-24 16:55:20,863 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/81325a8cd41e476386a744dc2e760e27 2023-03-24 16:55:20,886 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/2bf364b9198645c894d07b7577178087 2023-03-24 16:55:20,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:20,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676920990, max timestamp=1679676921003Num uniques 859; 2023-03-24 16:55:21,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:21,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676920990, max timestamp=1679676921003Num uniques 859; 2023-03-24 16:55:21,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:21,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676920990, max timestamp=1679676921003Num uniques 859; 2023-03-24 16:55:21,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:21,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676921022, max timestamp=1679676921055Num uniques 1068; 2023-03-24 16:55:21,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 16:55:21,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676921022, max timestamp=1679676921055Num uniques 1068; 2023-03-24 16:55:21,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676921022, max timestamp=1679676921055Num uniques 1068; 2023-03-24 16:55:21,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 16:55:21,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 16:55:21,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676921082, max timestamp=1679676921102Num uniques 1027; 2023-03-24 16:55:21,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:55:21,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676921082, max timestamp=1679676921102Num uniques 1027; 2023-03-24 16:55:21,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676921082, max timestamp=1679676921102Num uniques 1027; 2023-03-24 16:55:21,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:55:21,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676921104, max timestamp=1679676921154Num uniques 1036; 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676921104, max timestamp=1679676921154Num uniques 1036; 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676921104, max timestamp=1679676921154Num uniques 1036; 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 16:55:21,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676921184, max timestamp=1679676921201Num uniques 1145; 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-24 16:55:21,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676921184, max timestamp=1679676921201Num uniques 1145; 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679676921184, max timestamp=1679676921201Num uniques 1145; 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-24 16:55:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676981194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-24 16:55:21,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676921249, max timestamp=1679676921268Num uniques 1104; 2023-03-24 16:55:21,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676921249, max timestamp=1679676921268Num uniques 1104; 2023-03-24 16:55:21,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-24 16:55:21,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-24 16:55:21,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676921259, max timestamp=1679676921268Num uniques 1104; 2023-03-24 16:55:21,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 16:55:21,301 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/035c130c052d4dbbaccb20a6810dfe1e 2023-03-24 16:55:21,305 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/81325a8cd41e476386a744dc2e760e27 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27 2023-03-24 16:55:21,309 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27, entries=50, sequenceid=272, filesize=11.7 K 2023-03-24 16:55:21,309 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/2bf364b9198645c894d07b7577178087 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087 2023-03-24 16:55:21,312 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087, entries=50, sequenceid=272, filesize=7.1 K 2023-03-24 16:55:21,313 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/035c130c052d4dbbaccb20a6810dfe1e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e 2023-03-24 16:55:21,316 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e, entries=50, sequenceid=272, filesize=7.1 K 2023-03-24 16:55:21,317 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=136.42 KB/139690 for 0496a4cb656870861b4baf5a724e8ad6 in 514ms, sequenceid=272, compaction requested=true 2023-03-24 16:55:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:21,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=143.13 KB heapSize=405.77 KB 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:21,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:21,317 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:21,318 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,321 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d576e49[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,323 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,325 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,328 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,330 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,330 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,332 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,332 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676981327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,336 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,337 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,337 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676981331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676981333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1679676981348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676981349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,437 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:21,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:21,440 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:21,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:21,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676981435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52134 deadline: 1679676981438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676981438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:21,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,446 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,446 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,446 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd1e6f7[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,447 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,448 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,449 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,463 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,463 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:21,563 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:21,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:21,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:21,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:21,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:21,566 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:21,566 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:21,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:21,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,569 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:21,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:21,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:21,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:21,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:21,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,571 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,574 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52d7c473[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,575 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,579 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,581 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,583 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,583 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,586 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,591 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,592 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676981642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676981643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676981643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,692 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:21,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:21,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:21,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:21,695 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:21,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:21,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:21,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,697 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,698 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:21,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:21,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:21,699 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,699 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:21,699 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,700 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53eabf6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,704 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,710 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,713 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,713 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,721 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,722 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,733 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d142b53a7c8442f6a4bf111b68fce175 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d142b53a7c8442f6a4bf111b68fce175 2023-03-24 16:55:21,741 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/61ee50b1655a4c81af6d3376e1090acf 2023-03-24 16:55:21,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/244cb6891858461cbb0c9c2e53fb6bd4 2023-03-24 16:55:21,822 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:21,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:21,825 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:21,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:21,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:21,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,827 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:21,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:21,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:21,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:21,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,828 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,830 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@233da83e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,837 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,837 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,838 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,838 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,843 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,843 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,943 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:21,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:21,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:21,946 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:21,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:21,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:21,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,949 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:21,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676981947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1679676981948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:21,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:21,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:21,951 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,951 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:21,951 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:21,951 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,952 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235b14f0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:21,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:21,954 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:21,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676981949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,955 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:21,957 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:21,957 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:21,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,964 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:21,968 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:21,968 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:21,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:21,969 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:21,969 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:21,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:22,069 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:22,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:22,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:22,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:22,072 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:22,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:22,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:22,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:22,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:22,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:22,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,074 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:22,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:22,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:22,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:22,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:22,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:22,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:22,075 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:22,075 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:22,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:22,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:22,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:22,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:22,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc3a8b4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:22,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:22,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:22,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:22,084 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:22,086 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:22,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:22,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:22,086 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:22,086 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:22,088 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,088 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:22,088 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:22,090 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:22,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:22,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:22,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:22,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:22,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:22,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,094 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:22,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:22,094 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:22,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/57dc55256d6f4153af990572921c3957 2023-03-24 16:55:22,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/61ee50b1655a4c81af6d3376e1090acf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf 2023-03-24 16:55:22,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf, entries=250, sequenceid=395, filesize=39.2 K 2023-03-24 16:55:22,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/244cb6891858461cbb0c9c2e53fb6bd4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4 2023-03-24 16:55:22,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4, entries=150, sequenceid=395, filesize=11.9 K 2023-03-24 16:55:22,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/57dc55256d6f4153af990572921c3957 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957 2023-03-24 16:55:22,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957, entries=150, sequenceid=395, filesize=11.9 K 2023-03-24 16:55:22,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~405.72 KB/415456, currentSize=40.25 KB/41220 for 0496a4cb656870861b4baf5a724e8ad6 in 860ms, sequenceid=395, compaction requested=true 2023-03-24 16:55:22,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:22,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:22,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:22,177 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:22,177 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:22,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:22,178 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130216 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:22,178 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:22,178 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,179 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d7fc907a3757429489887111983f9f8d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=127.2 K 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56036 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:22,179 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,179 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ecd6d7805f1c4a2aaee42abca5830a57, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=54.7 K 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d7fc907a3757429489887111983f9f8d, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ecd6d7805f1c4a2aaee42abca5830a57, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 92f4129ee9894291a0374eebb5aaf53e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679676917859 2023-03-24 16:55:22,179 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3f47d84dd12143e8a91deb67085adb97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679676917859 2023-03-24 16:55:22,180 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a4d01872d7e241fea0d28470af8af341, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679676919174 2023-03-24 16:55:22,180 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 07ea8e09e25f4e59a98eafa32ee94afa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679676919174 2023-03-24 16:55:22,180 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 81325a8cd41e476386a744dc2e760e27, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679676919785 2023-03-24 16:55:22,180 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2bf364b9198645c894d07b7577178087, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679676919785 2023-03-24 16:55:22,180 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 61ee50b1655a4c81af6d3376e1090acf, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921298 2023-03-24 16:55:22,181 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 244cb6891858461cbb0c9c2e53fb6bd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921312 2023-03-24 16:55:22,190 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:22,193 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#362 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:22,194 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:22,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:22,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:22,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:22,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:22,204 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,207 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:22,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:22,208 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/85bf92f3a0334a349edf5af7119dae82 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/85bf92f3a0334a349edf5af7119dae82 2023-03-24 16:55:22,208 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,208 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:22,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:22,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:22,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:22,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:22,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:22,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:22,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:22,212 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,212 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:22,212 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:22,214 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 85bf92f3a0334a349edf5af7119dae82(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:22,214 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:22,214 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=11, startTime=1679676922177; duration=0sec 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:22,214 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:22,214 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:22,214 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:22,217 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56036 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:22,217 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:22,217 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:22,217 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/a8f777372e614fa5bfb1b2b8e269a8fb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=54.7 K 2023-03-24 16:55:22,218 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a8f777372e614fa5bfb1b2b8e269a8fb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679676917207 2023-03-24 16:55:22,218 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2c97329015c046a8a0db6733d5e59d24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679676917859 2023-03-24 16:55:22,218 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5fd65494ee964d4db6841ba0a62efd52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679676919174 2023-03-24 16:55:22,220 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 035c130c052d4dbbaccb20a6810dfe1e, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679676919785 2023-03-24 16:55:22,220 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 57dc55256d6f4153af990572921c3957, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921312 2023-03-24 16:55:22,233 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#365 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:22,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676922471, max timestamp=1679676922485Num uniques 859; 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676922471, max timestamp=1679676922485Num uniques 859; 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676922471, max timestamp=1679676922485Num uniques 859; 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:22,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676982527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676982528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676982529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,634 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245318a0fc2c3149508a4b8852eb6b62f8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245318a0fc2c3149508a4b8852eb6b62f8 2023-03-24 16:55:22,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676982630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,638 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/1cc3ce13f91c4a68a5c16d413ac13b75 2023-03-24 16:55:22,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676982636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,648 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/e94a6ce72dc54c68a14802ad7904f5f7 2023-03-24 16:55:22,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676982642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,651 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/e132a7700f274c5a9256658abe584f15 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e132a7700f274c5a9256658abe584f15 2023-03-24 16:55:22,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into e132a7700f274c5a9256658abe584f15(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:22,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:22,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=11, startTime=1679676922177; duration=0sec 2023-03-24 16:55:22,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676982839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676982844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:22,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676982851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,071 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/dc68592d9fc942459a0c0053735f94a4 2023-03-24 16:55:23,076 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/1cc3ce13f91c4a68a5c16d413ac13b75 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75 2023-03-24 16:55:23,079 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75, entries=150, sequenceid=405, filesize=25.4 K 2023-03-24 16:55:23,080 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/e94a6ce72dc54c68a14802ad7904f5f7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7 2023-03-24 16:55:23,083 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7, entries=150, sequenceid=405, filesize=11.9 K 2023-03-24 16:55:23,084 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/dc68592d9fc942459a0c0053735f94a4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4 2023-03-24 16:55:23,087 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4, entries=150, sequenceid=405, filesize=11.9 K 2023-03-24 16:55:23,088 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0496a4cb656870861b4baf5a724e8ad6 in 876ms, sequenceid=405, compaction requested=false 2023-03-24 16:55:23,088 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:23,088 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,090 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f52cba1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,097 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,097 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,097 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,097 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,097 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,099 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,099 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,099 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,100 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,104 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,104 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,104 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:23,148 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:23,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:23,161 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303246dd40faea1244948b95b713c7945adfa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246dd40faea1244948b95b713c7945adfa 2023-03-24 16:55:23,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676983162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676983168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52142 deadline: 1679676983169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,204 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,207 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,208 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,208 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,209 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,211 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,211 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,211 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,211 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,213 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e0e82d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,218 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,220 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,220 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,223 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,233 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,233 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:23,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676983272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676983272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676983274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,333 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,340 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,341 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,342 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,342 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a679e75[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,345 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,345 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,345 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,345 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,349 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,349 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,352 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,353 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,355 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,355 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,355 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1679676983356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,360 DEBUG [Thread-2463] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:23,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676983371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,377 DEBUG [Thread-2467] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:23,418 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/8fb98ba0c43c4bda918fe809062fb403 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/8fb98ba0c43c4bda918fe809062fb403 2023-03-24 16:55:23,422 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into 8fb98ba0c43c4bda918fe809062fb403(size=25.9 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 16:55:23,422 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:23,422 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=11, startTime=1679676922177; duration=1sec 2023-03-24 16:55:23,422 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:23,455 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,459 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,459 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,460 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,462 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,463 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,463 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,463 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58330185[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,470 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,473 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,473 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,475 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,477 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676983475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676983476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,480 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,480 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,481 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676983480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,567 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/b8deee58942f48eabc83ba427ad66d57 2023-03-24 16:55:23,580 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,583 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,586 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7195c6e1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,590 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,594 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,596 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,596 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,598 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,600 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,605 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,605 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,605 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,605 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,605 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,606 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,706 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,709 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,711 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,712 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,714 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,714 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c2f97e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,719 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,722 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,722 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,722 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,722 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,722 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,725 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,726 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,729 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,729 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:23,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,729 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,729 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,730 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676983782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676983782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:23,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676983786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,830 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,832 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,834 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,836 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,836 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,836 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,836 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,838 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e05bc5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,843 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,847 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,847 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,847 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,848 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,849 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,852 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:23,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,853 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,853 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,953 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:23,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:23,956 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:23,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,956 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:23,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:23,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,959 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,959 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:23,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:23,961 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,961 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:23,961 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:23,961 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:23,962 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711f5b55[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:23,964 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:23,964 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,965 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:23,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:23,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:23,969 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:23,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:23,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:23,969 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:23,969 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:23,972 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,972 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,972 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:23,972 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:23,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:23,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:23,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:23,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:23,975 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:23,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:23,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:23,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:23,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:23,976 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:23,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/ea564a3a754141c5ba7c5a10c1b662b1 2023-03-24 16:55:24,076 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:24,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:24,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:24,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:24,079 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:24,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:24,079 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:24,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,080 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,082 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:24,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:24,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:24,084 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,084 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:24,084 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:24,084 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:24,084 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,090 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,090 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:24,090 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee23959[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:24,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:24,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:24,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:24,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:24,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:24,097 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:24,100 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:24,100 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:24,100 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:24,100 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:24,100 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:24,102 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,102 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,102 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,103 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,108 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:24,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:24,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:24,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,109 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:24,109 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,209 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:24,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:24,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:24,213 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:24,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:24,213 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:24,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:24,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:24,224 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,224 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:24,224 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:24,224 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3efb3cf2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:24,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:24,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,231 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:24,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:24,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:24,236 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:24,236 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:24,236 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:24,236 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:24,236 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:24,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,244 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:24,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:24,245 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:24,245 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:24,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676984285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:24,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676984289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676984292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,345 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:24,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:24,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:24,356 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:24,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:24,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:24,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:24,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:24,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:24,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:24,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:24,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:24,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:24,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:24,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:24,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:24,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,384 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:24,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d76769b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:24,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:24,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,386 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:24,386 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,387 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:24,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/062b99e1207b4fb3aadcd4d90b73aa84 2023-03-24 16:55:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:24,388 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:24,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:24,393 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:24,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:24,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:24,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:24,393 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:24,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/b8deee58942f48eabc83ba427ad66d57 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57 2023-03-24 16:55:24,396 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,396 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,396 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:24,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:24,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:24,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57, entries=200, sequenceid=443, filesize=32.3 K 2023-03-24 16:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:24,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/ea564a3a754141c5ba7c5a10c1b662b1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1 2023-03-24 16:55:24,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1, entries=150, sequenceid=443, filesize=11.9 K 2023-03-24 16:55:24,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/062b99e1207b4fb3aadcd4d90b73aa84 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:24,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,408 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,408 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:24,408 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:24,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:24,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84, entries=150, sequenceid=443, filesize=11.9 K 2023-03-24 16:55:24,412 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 0496a4cb656870861b4baf5a724e8ad6 in 1264ms, sequenceid=443, compaction requested=true 2023-03-24 16:55:24,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:24,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:24,412 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:24,412 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:24,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:24,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:24,413 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:24,413 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:24,413 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:24,413 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:24,413 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,413 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,413 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/8fb98ba0c43c4bda918fe809062fb403, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=83.7 K 2023-03-24 16:55:24,413 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/85bf92f3a0334a349edf5af7119dae82, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=36.2 K 2023-03-24 16:55:24,414 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 8fb98ba0c43c4bda918fe809062fb403, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921312 2023-03-24 16:55:24,414 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 85bf92f3a0334a349edf5af7119dae82, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921312 2023-03-24 16:55:24,415 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1cc3ce13f91c4a68a5c16d413ac13b75, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679676921326 2023-03-24 16:55:24,415 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e94a6ce72dc54c68a14802ad7904f5f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679676921326 2023-03-24 16:55:24,416 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b8deee58942f48eabc83ba427ad66d57, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:24,416 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ea564a3a754141c5ba7c5a10c1b662b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:24,429 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:24,433 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#371 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:24,508 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:24,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:24,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:24,511 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:24,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:24,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:24,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,513 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,513 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:24,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:24,513 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:24,514 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:24,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:24,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:24,515 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,515 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:24,515 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 16:55:24,515 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:24,515 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:24,515 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:24,516 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:24,516 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:24,516 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:24,531 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240af0982689f04144b82a0161a1b99e94 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240af0982689f04144b82a0161a1b99e94 2023-03-24 16:55:24,843 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/3947179c7099462a8132b9128f6930db as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3947179c7099462a8132b9128f6930db 2023-03-24 16:55:24,847 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 3947179c7099462a8132b9128f6930db(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:24,847 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:24,847 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=13, startTime=1679676924412; duration=0sec 2023-03-24 16:55:24,847 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:24,847 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:24,848 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:24,848 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:24,848 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:24,848 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e132a7700f274c5a9256658abe584f15, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=36.2 K 2023-03-24 16:55:24,848 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e132a7700f274c5a9256658abe584f15, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679676921312 2023-03-24 16:55:24,849 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting dc68592d9fc942459a0c0053735f94a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679676921326 2023-03-24 16:55:24,849 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 062b99e1207b4fb3aadcd4d90b73aa84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:24,854 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#374 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:24,938 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e1170d02d8b2414889858ccc2cab2922 2023-03-24 16:55:25,264 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/073dc84f268f41d09b50df63ac4fea2d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/073dc84f268f41d09b50df63ac4fea2d 2023-03-24 16:55:25,268 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into 073dc84f268f41d09b50df63ac4fea2d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:25,268 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:25,268 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=13, startTime=1679676924412; duration=0sec 2023-03-24 16:55:25,268 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:25,279 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/128abfc4712749fd8f802f620efa0090 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/128abfc4712749fd8f802f620efa0090 2023-03-24 16:55:25,283 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into 128abfc4712749fd8f802f620efa0090(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:25,283 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:25,283 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=13, startTime=1679676924412; duration=0sec 2023-03-24 16:55:25,283 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:25,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925311, max timestamp=1679676925329Num uniques 859; 2023-03-24 16:55:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925311, max timestamp=1679676925329Num uniques 859; 2023-03-24 16:55:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925311, max timestamp=1679676925329Num uniques 859; 2023-03-24 16:55:25,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,347 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/608d9b1e34554372abfcc42122637060 2023-03-24 16:55:25,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676985354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676985356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676985357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,372 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/31deb712ea414997bdd78db6f859beec 2023-03-24 16:55:25,375 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e1170d02d8b2414889858ccc2cab2922 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922 2023-03-24 16:55:25,380 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922, entries=150, sequenceid=452, filesize=25.4 K 2023-03-24 16:55:25,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/608d9b1e34554372abfcc42122637060 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060 2023-03-24 16:55:25,386 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060, entries=150, sequenceid=452, filesize=11.9 K 2023-03-24 16:55:25,387 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/31deb712ea414997bdd78db6f859beec as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec 2023-03-24 16:55:25,390 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec, entries=150, sequenceid=452, filesize=11.9 K 2023-03-24 16:55:25,391 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 0496a4cb656870861b4baf5a724e8ad6 in 875ms, sequenceid=452, compaction requested=false 2023-03-24 16:55:25,391 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:25,391 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:25,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:25,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:25,395 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,395 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6385193[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:25,397 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:25,397 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:25,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:25,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:25,407 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:25,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:25,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:25,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:25,407 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:25,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:25,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,416 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,416 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:25,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676925353, max timestamp=1679676925461Num uniques 1118; 2023-03-24 16:55:25,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-24 16:55:25,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676925353, max timestamp=1679676925461Num uniques 1118; 2023-03-24 16:55:25,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-24 16:55:25,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=69.33 KB heapSize=188.95 KB 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:25,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676925353, max timestamp=1679676925461Num uniques 1118; 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032497b42be37b3c4819a93382c91876f6d6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032497b42be37b3c4819a93382c91876f6d6 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925487, max timestamp=1679676925506Num uniques 859; 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925487, max timestamp=1679676925506Num uniques 859; 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676925487, max timestamp=1679676925506Num uniques 859; 2023-03-24 16:55:25,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:55:25,517 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:25,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:25,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:25,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:25,519 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:25,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:25,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:25,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:25,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:25,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:25,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:25,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:25,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:25,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:25,522 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,522 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:25,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:25,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:25,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,528 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,528 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:25,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@392f88fc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:25,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:25,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:25,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:25,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:25,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:25,533 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:25,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:25,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:25,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:25,534 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:25,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:25,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,540 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:25,540 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:25,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676925533, max timestamp=1679676925550Num uniques 1118; 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679676925533, max timestamp=1679676925550Num uniques 1118; 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679676925533, max timestamp=1679676925550Num uniques 1118; 2023-03-24 16:55:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-03-24 16:55:25,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676925573, max timestamp=1679676925591Num uniques 1177; 2023-03-24 16:55:25,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-03-24 16:55:25,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676925573, max timestamp=1679676925591Num uniques 1077; 2023-03-24 16:55:25,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679676925573, max timestamp=1679676925591Num uniques 1077; 2023-03-24 16:55:25,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:55:25,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 16:55:25,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676925616, max timestamp=1679676925626Num uniques 1136; 2023-03-24 16:55:25,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-24 16:55:25,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676925616, max timestamp=1679676925626Num uniques 1136; 2023-03-24 16:55:25,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-24 16:55:25,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679676925616, max timestamp=1679676925626Num uniques 1136; 2023-03-24 16:55:25,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-24 16:55:25,641 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:25,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:25,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:25,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:25,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:25,643 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:25,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:25,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:25,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,645 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:25,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:25,646 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,647 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:25,647 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:25,647 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:25,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:25,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:25,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:25,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed43f87[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:25,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:25,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:25,651 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,651 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:25,653 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:25,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:25,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:25,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:25,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:25,656 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:25,656 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:25,657 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,657 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,658 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,659 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:25,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:25,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,662 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:25,662 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:25,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676925647, max timestamp=1679676925671Num uniques 1145; 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676925647, max timestamp=1679676925671Num uniques 1145; 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676925647, max timestamp=1679676925671Num uniques 1145; 2023-03-24 16:55:25,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-24 16:55:25,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676925727, max timestamp=1679676925746Num uniques 1154; 2023-03-24 16:55:25,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-24 16:55:25,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676925727, max timestamp=1679676925746Num uniques 1154; 2023-03-24 16:55:25,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679676925727, max timestamp=1679676925746Num uniques 1154; 2023-03-24 16:55:25,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-24 16:55:25,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-24 16:55:25,762 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:25,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:25,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:25,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:25,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:25,765 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:25,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:25,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:25,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:25,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:25,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:25,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:25,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:25,769 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:25,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,771 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:25,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60284868[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:25,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:25,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,774 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:25,774 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:25,774 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:25,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:25,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:25,781 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:25,781 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:25,781 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:25,781 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:25,781 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:25,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,784 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676925758, max timestamp=1679676925784Num uniques 1113; 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676925758, max timestamp=1679676925784Num uniques 1113; 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 16:55:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676925758, max timestamp=1679676925784Num uniques 1113; 2023-03-24 16:55:25,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 16:55:25,788 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,790 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:25,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,790 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:25,790 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:25,790 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676985823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676925809, max timestamp=1679676925822Num uniques 1072; 2023-03-24 16:55:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676925809, max timestamp=1679676925822Num uniques 1072; 2023-03-24 16:55:25,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676925809, max timestamp=1679676925822Num uniques 1072; 2023-03-24 16:55:25,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 16:55:25,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 16:55:25,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 16:55:25,890 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:25,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:25,894 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:25,894 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,894 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:25,894 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:25,895 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,895 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,896 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/faaf7bc86fe943d0a0e2950a56b96370 2023-03-24 16:55:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:25,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679676925885, max timestamp=1679676925898Num uniques 1031; 2023-03-24 16:55:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-24 16:55:25,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679676925885, max timestamp=1679676925898Num uniques 1031; 2023-03-24 16:55:25,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-24 16:55:25,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679676925885, max timestamp=1679676925898Num uniques 1031; 2023-03-24 16:55:25,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-24 16:55:25,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:25,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:25,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:25,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:25,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:25,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:25,911 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:25,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:25,914 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,914 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/46d86eb3706f45029311cc589466b7e0 2023-03-24 16:55:25,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:25,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24582937[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:25,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:25,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:25,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:25,918 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:25,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:25,920 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:25,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:25,922 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:25,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:25,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:25,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:25,922 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:25,924 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:25,924 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,924 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:25,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/4ba17c1ddf3f4ae1853a340bf0e575f3 2023-03-24 16:55:25,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/faaf7bc86fe943d0a0e2950a56b96370 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370 2023-03-24 16:55:25,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:25,930 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:25,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:25,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:25,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:25,931 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,931 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:25,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:25,931 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:25,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:25,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370, entries=200, sequenceid=493, filesize=32.3 K 2023-03-24 16:55:25,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/46d86eb3706f45029311cc589466b7e0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0 2023-03-24 16:55:25,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0, entries=150, sequenceid=493, filesize=11.9 K 2023-03-24 16:55:25,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/4ba17c1ddf3f4ae1853a340bf0e575f3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3 2023-03-24 16:55:25,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3, entries=150, sequenceid=493, filesize=11.9 K 2023-03-24 16:55:25,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=114.05 KB/116790 for 0496a4cb656870861b4baf5a724e8ad6 in 486ms, sequenceid=493, compaction requested=true 2023-03-24 16:55:25,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:25,951 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:25,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:25,952 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:25,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:25,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:25,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=127.47 KB heapSize=363 KB 2023-03-24 16:55:25,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:25,953 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:25,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,953 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:25,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:25,953 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:25,953 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,953 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:25,953 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3947179c7099462a8132b9128f6930db, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=36.3 K 2023-03-24 16:55:25,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,953 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:25,953 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/128abfc4712749fd8f802f620efa0090, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=83.8 K 2023-03-24 16:55:25,954 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3947179c7099462a8132b9128f6930db, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:25,954 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 128abfc4712749fd8f802f620efa0090, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:25,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:25,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:25,955 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 608d9b1e34554372abfcc42122637060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676923161 2023-03-24 16:55:25,955 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e1170d02d8b2414889858ccc2cab2922, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676923161 2023-03-24 16:55:25,956 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 46d86eb3706f45029311cc589466b7e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925355 2023-03-24 16:55:25,956 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting faaf7bc86fe943d0a0e2950a56b96370, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925353 2023-03-24 16:55:25,969 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676985975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676985976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:25,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676985980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,011 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#380 average throughput is 0.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:26,031 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324449b7cba8d7c4f208200845c1ae6275e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324449b7cba8d7c4f208200845c1ae6275e 2023-03-24 16:55:26,031 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:26,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:26,037 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:26,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:26,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:26,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:26,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:26,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:26,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,043 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:26,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:26,046 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:26,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:26,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:26,047 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:26,048 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=666, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/f995b753576c4a0c9e0d50fa53d0f9ae 2023-03-24 16:55:26,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,049 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,049 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:26,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61752d28[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:26,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:26,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,052 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:26,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:26,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:26,067 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:26,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:26,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:26,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:26,067 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:26,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f4375de3829e4a8788ff8958e270ba0c 2023-03-24 16:55:26,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,069 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,076 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,077 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,078 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:26,078 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:26,081 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/bfe2729792724061baedf85181c88c64 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bfe2729792724061baedf85181c88c64 2023-03-24 16:55:26,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/d5b2125f1f764ff2a350a321e22712ef 2023-03-24 16:55:26,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/f995b753576c4a0c9e0d50fa53d0f9ae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae 2023-03-24 16:55:26,088 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into bfe2729792724061baedf85181c88c64(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:26,089 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:26,089 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=13, startTime=1679676925951; duration=0sec 2023-03-24 16:55:26,089 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:26,089 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:26,090 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:26,090 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:26,090 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,090 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/073dc84f268f41d09b50df63ac4fea2d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=36.3 K 2023-03-24 16:55:26,091 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 073dc84f268f41d09b50df63ac4fea2d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676922519 2023-03-24 16:55:26,091 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 31deb712ea414997bdd78db6f859beec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676923161 2023-03-24 16:55:26,091 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4ba17c1ddf3f4ae1853a340bf0e575f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925355 2023-03-24 16:55:26,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae, entries=350, sequenceid=666, filesize=52.9 K 2023-03-24 16:55:26,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f4375de3829e4a8788ff8958e270ba0c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c 2023-03-24 16:55:26,097 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#385 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676986087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676986088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676986088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c, entries=150, sequenceid=666, filesize=11.9 K 2023-03-24 16:55:26,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/d5b2125f1f764ff2a350a321e22712ef as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef 2023-03-24 16:55:26,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef, entries=150, sequenceid=666, filesize=11.9 K 2023-03-24 16:55:26,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~362.95 KB/371664, currentSize=53.67 KB/54960 for 0496a4cb656870861b4baf5a724e8ad6 in 155ms, sequenceid=666, compaction requested=false 2023-03-24 16:55:26,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:26,113 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/914461a4817a452d87bab0ac0f739854 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/914461a4817a452d87bab0ac0f739854 2023-03-24 16:55:26,117 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into 914461a4817a452d87bab0ac0f739854(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:26,117 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:26,117 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=13, startTime=1679676925952; duration=0sec 2023-03-24 16:55:26,117 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:26,178 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:26,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:26,181 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:26,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:26,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:26,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,183 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:26,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:26,185 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:26,202 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303243a7aa82c50f347b2b9859dadcf70a7ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a7aa82c50f347b2b9859dadcf70a7ab 2023-03-24 16:55:26,207 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=679, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e14d01d20df44f5fab3cdb4c0057f9eb 2023-03-24 16:55:26,220 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f0bdfa3590c0460a8f70e4dc61a9b15f 2023-03-24 16:55:26,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:26,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:26,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676986328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676926323, max timestamp=1679676926338Num uniques 859; 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676926323, max timestamp=1679676926338Num uniques 859; 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676926323, max timestamp=1679676926338Num uniques 859; 2023-03-24 16:55:26,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:26,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676986381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676986382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,427 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/197f3b536f80415d8f7b32e8151c6d4c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/197f3b536f80415d8f7b32e8151c6d4c 2023-03-24 16:55:26,430 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 197f3b536f80415d8f7b32e8151c6d4c(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:26,430 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:26,430 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=13, startTime=1679676925951; duration=0sec 2023-03-24 16:55:26,431 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:26,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676986440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676986486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676986488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,628 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/2d51b08bda5f4056b58c8a78a189575b 2023-03-24 16:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e14d01d20df44f5fab3cdb4c0057f9eb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb 2023-03-24 16:55:26,636 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb, entries=150, sequenceid=679, filesize=25.4 K 2023-03-24 16:55:26,636 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/f0bdfa3590c0460a8f70e4dc61a9b15f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f 2023-03-24 16:55:26,639 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f, entries=150, sequenceid=679, filesize=11.9 K 2023-03-24 16:55:26,640 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/2d51b08bda5f4056b58c8a78a189575b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b 2023-03-24 16:55:26,642 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b, entries=150, sequenceid=679, filesize=11.9 K 2023-03-24 16:55:26,643 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0496a4cb656870861b4baf5a724e8ad6 in 458ms, sequenceid=679, compaction requested=true 2023-03-24 16:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:26,643 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:26,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:26,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:26,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:26,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,645 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,645 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:26,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e220e86[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:26,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:26,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,647 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:26,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:26,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:26,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:26,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:26,653 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:26,653 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:26,653 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:26,653 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:26,653 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:26,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,662 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,662 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:26,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:26,663 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:26,663 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:26,666 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240bcfd3b1478b43159234e5a56f344b15 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bcfd3b1478b43159234e5a56f344b15 2023-03-24 16:55:26,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676986695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676986697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676986703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,763 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:26,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:26,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:26,766 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:26,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:26,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:26,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,767 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,774 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,774 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:26,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:26,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:26,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:26,784 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,784 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:26,784 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:26,784 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:26,785 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ceb6e22[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:26,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:26,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:26,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,788 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:26,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,791 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:26,794 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:26,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:26,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:26,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:26,794 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:26,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,798 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:26,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:26,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:26,803 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:26,803 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:26,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676986805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676986806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,904 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:26,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:26,907 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:26,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:26,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:26,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:26,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:26,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:26,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:26,910 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:26,912 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@591ff919[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:26,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:26,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,916 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:26,916 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:26,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:26,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:26,920 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:26,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:26,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:26,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:26,920 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:26,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,922 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:26,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:26,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:26,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:26,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:26,926 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,926 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,926 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:26,927 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676987005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676987012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676987012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,026 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,029 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,030 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,030 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,033 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,034 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,034 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,034 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,035 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,036 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11cb837f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,038 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,038 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,038 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,042 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,042 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,046 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:27,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,051 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,051 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,052 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,051 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,052 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,052 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,071 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=716, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/32678adce4f2477d8a6a42f0e405835e 2023-03-24 16:55:27,152 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,155 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,156 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,156 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,158 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,158 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,158 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,158 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,161 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,161 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f93f9fe[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,163 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,167 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,167 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:27,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,174 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,174 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,274 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,277 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,279 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,281 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,281 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,281 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,281 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ee0e50[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,285 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,293 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,297 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,298 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,298 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,300 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:27,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,306 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,306 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676987329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676987329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1679676987380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,387 DEBUG [Thread-2463] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:27,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52188 deadline: 1679676987400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,406 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,406 DEBUG [Thread-2467] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,409 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,409 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,409 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,409 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e9fe765[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,419 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,421 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,425 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,425 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,425 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,430 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,431 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,431 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,431 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/049bf41e1ce04f6194cc785ebb68a384 2023-03-24 16:55:27,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1679676987512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,531 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,534 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,541 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,541 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,541 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,541 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,543 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138257b2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,546 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,548 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,550 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,550 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,556 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,558 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,559 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,559 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,659 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,664 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,667 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,668 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,668 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,668 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,668 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13de0035[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,677 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,677 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,683 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,685 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,686 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,686 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,786 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,789 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,790 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,790 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,791 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,796 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,798 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4a1090[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,800 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,803 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,805 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,807 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,812 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,813 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52134 deadline: 1679676987841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:27,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52142 deadline: 1679676987842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/de9685ad6d8a434e9306f3a10ff61780 2023-03-24 16:55:27,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/32678adce4f2477d8a6a42f0e405835e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e 2023-03-24 16:55:27,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e, entries=200, sequenceid=716, filesize=32.3 K 2023-03-24 16:55:27,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/049bf41e1ce04f6194cc785ebb68a384 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384 2023-03-24 16:55:27,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384, entries=150, sequenceid=716, filesize=11.9 K 2023-03-24 16:55:27,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/de9685ad6d8a434e9306f3a10ff61780 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780 2023-03-24 16:55:27,913 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:27,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:27,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:27,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780, entries=150, sequenceid=716, filesize=11.9 K 2023-03-24 16:55:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:27,916 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:27,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:27,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:27,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,916 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:27,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:27,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:27,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:27,920 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:27,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0496a4cb656870861b4baf5a724e8ad6 in 1272ms, sequenceid=716, compaction requested=true 2023-03-24 16:55:27,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:27,921 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:27,921 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:27,922 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:27,922 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/B is initiating minor compaction (all files) 2023-03-24 16:55:27,922 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 140108 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:27,922 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/B in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,922 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/A is initiating minor compaction (all files) 2023-03-24 16:55:27,922 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/197f3b536f80415d8f7b32e8151c6d4c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=48.3 K 2023-03-24 16:55:27,922 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/A in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:27,922 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bfe2729792724061baedf85181c88c64, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=136.8 K 2023-03-24 16:55:27,922 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 197f3b536f80415d8f7b32e8151c6d4c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925355 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting bfe2729792724061baedf85181c88c64, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925355 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f4375de3829e4a8788ff8958e270ba0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1679676925923 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f995b753576c4a0c9e0d50fa53d0f9ae, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1679676925923 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f0bdfa3590c0460a8f70e4dc61a9b15f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1679676925972 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e14d01d20df44f5fab3cdb4c0057f9eb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1679676925972 2023-03-24 16:55:27,923 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 049bf41e1ce04f6194cc785ebb68a384, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679676926379 2023-03-24 16:55:27,924 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 32678adce4f2477d8a6a42f0e405835e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679676926378 2023-03-24 16:55:27,931 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#B#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:27,934 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3acf7853[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:27,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:27,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,939 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:27,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:27,940 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:27,943 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:27,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:27,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:27,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:27,943 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:27,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:27,946 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,946 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:27,950 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:27,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:27,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:27,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:27,951 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:27,951 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:27,950 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:27,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:27,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:27,954 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#A#compaction#393 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:28,051 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:28,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:28,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:28,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:28,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:28,054 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:28,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:28,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:28,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:28,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:28,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:28,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:28,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:28,057 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:28,058 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,071 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a9c23751bf88443c9b21852b12f05c0b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a9c23751bf88443c9b21852b12f05c0b 2023-03-24 16:55:28,077 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=726, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e9fa7f2ddc2e4eee811f9b1392b18504 2023-03-24 16:55:28,357 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/44047f6a93e448d8805d325d222b3e3a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/44047f6a93e448d8805d325d222b3e3a 2023-03-24 16:55:28,361 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/B of 0496a4cb656870861b4baf5a724e8ad6 into 44047f6a93e448d8805d325d222b3e3a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:28,362 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:28,362 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/B, priority=12, startTime=1679676927921; duration=0sec 2023-03-24 16:55:28,362 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:28,362 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:28,363 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:28,364 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 0496a4cb656870861b4baf5a724e8ad6/C is initiating minor compaction (all files) 2023-03-24 16:55:28,364 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0496a4cb656870861b4baf5a724e8ad6/C in TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:28,364 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/914461a4817a452d87bab0ac0f739854, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp, totalSize=48.3 K 2023-03-24 16:55:28,364 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 914461a4817a452d87bab0ac0f739854, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679676925355 2023-03-24 16:55:28,364 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d5b2125f1f764ff2a350a321e22712ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1679676925923 2023-03-24 16:55:28,365 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2d51b08bda5f4056b58c8a78a189575b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1679676925972 2023-03-24 16:55:28,365 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting de9685ad6d8a434e9306f3a10ff61780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679676926379 2023-03-24 16:55:28,373 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 0496a4cb656870861b4baf5a724e8ad6#C#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:28,388 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/004bd6310eeb44fd84362e3bef29323c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/004bd6310eeb44fd84362e3bef29323c 2023-03-24 16:55:28,392 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/C of 0496a4cb656870861b4baf5a724e8ad6 into 004bd6310eeb44fd84362e3bef29323c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:28,392 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:28,392 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/C, priority=12, startTime=1679676927921; duration=0sec 2023-03-24 16:55:28,392 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:28,486 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/1670f7523d454026885e0076ed2e0a31 2023-03-24 16:55:28,495 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/ae702f8ed03448989b4515c33d508e37 2023-03-24 16:55:28,498 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/e9fa7f2ddc2e4eee811f9b1392b18504 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e9fa7f2ddc2e4eee811f9b1392b18504 2023-03-24 16:55:28,501 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e9fa7f2ddc2e4eee811f9b1392b18504, entries=150, sequenceid=726, filesize=25.4 K 2023-03-24 16:55:28,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/1670f7523d454026885e0076ed2e0a31 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/1670f7523d454026885e0076ed2e0a31 2023-03-24 16:55:28,505 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/1670f7523d454026885e0076ed2e0a31, entries=150, sequenceid=726, filesize=11.9 K 2023-03-24 16:55:28,505 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/ae702f8ed03448989b4515c33d508e37 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ae702f8ed03448989b4515c33d508e37 2023-03-24 16:55:28,518 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ae702f8ed03448989b4515c33d508e37, entries=150, sequenceid=726, filesize=11.9 K 2023-03-24 16:55:28,518 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 0496a4cb656870861b4baf5a724e8ad6 in 461ms, sequenceid=726, compaction requested=false 2023-03-24 16:55:28,518 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:28,518 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:28,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf1877f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:28,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:28,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:28,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:28,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:28,533 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:28,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:28,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:28,533 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:28,533 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:28,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:28,535 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:28,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:28,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:28,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:28,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:28,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:28,540 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:28,540 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:28,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:28,640 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:28,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:28,644 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:28,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:28,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:28,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:28,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:28,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:28,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:28,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:28,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:28,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:28,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:28,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:28,656 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:28,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:28,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. started... 2023-03-24 16:55:28,656 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-24 16:55:28,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:28,656 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,657 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:28,657 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,657 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:28,657 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:28,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. as already flushing 2023-03-24 16:55:28,803 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/f35b087cf5e84d098c4e5b82f8fcf1a2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f35b087cf5e84d098c4e5b82f8fcf1a2 2023-03-24 16:55:28,807 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0496a4cb656870861b4baf5a724e8ad6/A of 0496a4cb656870861b4baf5a724e8ad6 into f35b087cf5e84d098c4e5b82f8fcf1a2(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:28,807 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:28,807 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6., storeName=0496a4cb656870861b4baf5a724e8ad6/A, priority=12, startTime=1679676927921; duration=0sec 2023-03-24 16:55:28,807 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:28,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676928845, max timestamp=1679676928878Num uniques 859; 2023-03-24 16:55:28,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676928845, max timestamp=1679676928878Num uniques 859; 2023-03-24 16:55:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676928845, max timestamp=1679676928878Num uniques 859; 2023-03-24 16:55:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679676928908, max timestamp=1679676928926Num uniques 1018; 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679676928908, max timestamp=1679676928926Num uniques 1018; 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679676928908, max timestamp=1679676928926Num uniques 1018; 2023-03-24 16:55:28,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676928967, max timestamp=1679676928985Num uniques 1177; 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676928967, max timestamp=1679676928985Num uniques 1177; 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676928967, max timestamp=1679676928985Num uniques 1177; 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 16:55:28,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 16:55:29,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10796, totalHeapSize=24784, min timestamp=1679676929017, max timestamp=1679676929053Num uniques 1036; 2023-03-24 16:55:29,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10796, totalHeapSize=24784, min timestamp=1679676929017, max timestamp=1679676929053Num uniques 1036; 2023-03-24 16:55:29,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=10796, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-24 16:55:29,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679676929017, max timestamp=1679676929053Num uniques 1036; 2023-03-24 16:55:29,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-24 16:55:29,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=10796, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-24 16:55:29,100 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324369993de42c64d30a3bc97ead29725e9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324369993de42c64d30a3bc97ead29725e9 2023-03-24 16:55:29,117 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=733, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/242de6a01b2a48b09646101274a8abcc 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676929106, max timestamp=1679676929134Num uniques 1095; 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676929106, max timestamp=1679676929134Num uniques 1095; 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679676929106, max timestamp=1679676929134Num uniques 1009; 2023-03-24 16:55:29,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-03-24 16:55:29,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676929173, max timestamp=1679676929195Num uniques 1154; 2023-03-24 16:55:29,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-03-24 16:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676929173, max timestamp=1679676929195Num uniques 1154; 2023-03-24 16:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-03-24 16:55:29,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676929173, max timestamp=1679676929195Num uniques 1154; 2023-03-24 16:55:29,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-03-24 16:55:29,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676929230, max timestamp=1679676929245Num uniques 1013; 2023-03-24 16:55:29,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676929230, max timestamp=1679676929245Num uniques 1013; 2023-03-24 16:55:29,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 16:55:29,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679676929230, max timestamp=1679676929245Num uniques 1013; 2023-03-24 16:55:29,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 16:55:29,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 16:55:29,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676929284, max timestamp=1679676929297Num uniques 1072; 2023-03-24 16:55:29,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 16:55:29,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676929284, max timestamp=1679676929297Num uniques 1072; 2023-03-24 16:55:29,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 16:55:29,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679676929284, max timestamp=1679676929297Num uniques 1072; 2023-03-24 16:55:29,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 16:55:29,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679676929336, max timestamp=1679676929357Num uniques 1081; 2023-03-24 16:55:29,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-24 16:55:29,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679676929336, max timestamp=1679676929357Num uniques 1081; 2023-03-24 16:55:29,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-24 16:55:29,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679676929336, max timestamp=1679676929357Num uniques 1081; 2023-03-24 16:55:29,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-24 16:55:29,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679676929380, max timestamp=1679676929396Num uniques 1090; 2023-03-24 16:55:29,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679676929380, max timestamp=1679676929396Num uniques 1090; 2023-03-24 16:55:29,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-24 16:55:29,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-24 16:55:29,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679676929388, max timestamp=1679676929396Num uniques 1090; 2023-03-24 16:55:29,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-24 16:55:29,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676929420, max timestamp=1679676929440Num uniques 1149; 2023-03-24 16:55:29,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-24 16:55:29,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676929420, max timestamp=1679676929440Num uniques 1149; 2023-03-24 16:55:29,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679676929420, max timestamp=1679676929440Num uniques 1049; 2023-03-24 16:55:29,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-24 16:55:29,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679676929465, max timestamp=1679676929479Num uniques 1058; 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679676929465, max timestamp=1679676929479Num uniques 1058; 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679676929465, max timestamp=1679676929479Num uniques 1058; 2023-03-24 16:55:29,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679676929490, max timestamp=1679676929523Num uniques 1067; 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679676929490, max timestamp=1679676929523Num uniques 1067; 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679676929490, max timestamp=1679676929523Num uniques 1067; 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-24 16:55:29,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-24 16:55:29,547 DEBUG [Thread-2465] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:62195 2023-03-24 16:55:29,547 DEBUG [Thread-2465] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,547 DEBUG [Thread-2476] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:62195 2023-03-24 16:55:29,547 DEBUG [Thread-2476] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,548 DEBUG [Thread-2461] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:62195 2023-03-24 16:55:29,548 DEBUG [Thread-2461] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,551 DEBUG [Thread-2472] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:62195 2023-03-24 16:55:29,551 DEBUG [Thread-2472] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,551 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/59af51de83c041d584c71c9c4110fc4e 2023-03-24 16:55:29,552 DEBUG [Thread-2474] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:62195 2023-03-24 16:55:29,552 DEBUG [Thread-2474] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,552 DEBUG [Thread-2469] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:62195 2023-03-24 16:55:29,552 DEBUG [Thread-2469] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,557 DEBUG [Thread-2480] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:62195 2023-03-24 16:55:29,557 DEBUG [Thread-2480] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,557 DEBUG [Thread-2478] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:62195 2023-03-24 16:55:29,558 DEBUG [Thread-2478] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:29,561 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/76529d5b3efd4802af26260b93ce82d2 2023-03-24 16:55:29,565 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/242de6a01b2a48b09646101274a8abcc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/242de6a01b2a48b09646101274a8abcc 2023-03-24 16:55:29,567 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/242de6a01b2a48b09646101274a8abcc, entries=100, sequenceid=733, filesize=18.6 K 2023-03-24 16:55:29,568 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/59af51de83c041d584c71c9c4110fc4e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/59af51de83c041d584c71c9c4110fc4e 2023-03-24 16:55:29,570 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/59af51de83c041d584c71c9c4110fc4e, entries=100, sequenceid=733, filesize=9.5 K 2023-03-24 16:55:29,570 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/76529d5b3efd4802af26260b93ce82d2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/76529d5b3efd4802af26260b93ce82d2 2023-03-24 16:55:29,573 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/76529d5b3efd4802af26260b93ce82d2, entries=100, sequenceid=733, filesize=9.5 K 2023-03-24 16:55:29,573 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=114.05 KB/116790 for 0496a4cb656870861b4baf5a724e8ad6 in 917ms, sequenceid=733, compaction requested=true 2023-03-24 16:55:29,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:29,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:29,752 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58cdf79c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,756 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:29,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:29,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:29,760 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:29,760 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:29,760 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:29,760 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:29,760 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:29,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:29,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:29,763 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:29,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:29,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:29,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:29,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:29,766 DEBUG [Thread-2471] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:29,766 DEBUG [Thread-2471] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:29,766 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:29,866 DEBUG [Thread-2471] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:37,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:37,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=120.76 KB heapSize=339.70 KB 2023-03-24 16:55:37,440 DEBUG [Thread-2463] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:62195 2023-03-24 16:55:37,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:37,440 DEBUG [Thread-2463] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:37,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:37,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:37,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:37,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:37,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:37,489 DEBUG [Thread-2467] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:62195 2023-03-24 16:55:37,489 DEBUG [Thread-2467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 241 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 225 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 263 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1915 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5745 rows 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1909 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5727 rows 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1902 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5706 rows 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1909 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5727 rows 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1904 2023-03-24 16:55:37,489 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5712 rows 2023-03-24 16:55:37,490 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:55:37,490 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:62195 2023-03-24 16:55:37,490 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:55:37,491 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:55:37,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:55:37,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:37,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:37,496 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676937496"}]},"ts":"1679676937496"} 2023-03-24 16:55:37,497 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:55:37,498 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:55:37,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, UNASSIGN}] 2023-03-24 16:55:37,500 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, UNASSIGN 2023-03-24 16:55:37,500 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:37,500 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676937500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676937500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676937500"}]},"ts":"1679676937500"} 2023-03-24 16:55:37,501 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:37,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:37,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:37,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0496a4cb656870861b4baf5a724e8ad6, disabling compactions & flushes 2023-03-24 16:55:37,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:37,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:37,852 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241e08bc1b94f745c58b53f3d2b71f2287 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241e08bc1b94f745c58b53f3d2b71f2287 2023-03-24 16:55:38,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:38,256 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=974, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/9eb6628dd9f34ce7a2355ac60afdf8ab 2023-03-24 16:55:38,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:38,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/c1237864d06d4eada59c35ac6251aeb3 2023-03-24 16:55:39,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/3bfa2754758f4df8b8e50c1e44d422fb 2023-03-24 16:55:39,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/9eb6628dd9f34ce7a2355ac60afdf8ab as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9eb6628dd9f34ce7a2355ac60afdf8ab 2023-03-24 16:55:39,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9eb6628dd9f34ce7a2355ac60afdf8ab, entries=150, sequenceid=974, filesize=25.4 K 2023-03-24 16:55:39,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/c1237864d06d4eada59c35ac6251aeb3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/c1237864d06d4eada59c35ac6251aeb3 2023-03-24 16:55:39,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/c1237864d06d4eada59c35ac6251aeb3, entries=150, sequenceid=974, filesize=11.9 K 2023-03-24 16:55:39,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/3bfa2754758f4df8b8e50c1e44d422fb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/3bfa2754758f4df8b8e50c1e44d422fb 2023-03-24 16:55:39,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/3bfa2754758f4df8b8e50c1e44d422fb, entries=150, sequenceid=974, filesize=11.9 K 2023-03-24 16:55:39,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~339.66 KB/347808, currentSize=6.71 KB/6870 for 0496a4cb656870861b4baf5a724e8ad6 in 1650ms, sequenceid=974, compaction requested=true 2023-03-24 16:55:39,090 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:39,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:39,090 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. because compaction request was cancelled 2023-03-24 16:55:39,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:39,090 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:39,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. after waiting 0 ms 2023-03-24 16:55:39,090 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:39,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:39,090 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. because compaction request was cancelled 2023-03-24 16:55:39,090 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:39,090 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. because compaction request was cancelled 2023-03-24 16:55:39,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0496a4cb656870861b4baf5a724e8ad6 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=A 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=B 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0496a4cb656870861b4baf5a724e8ad6, store=C 2023-03-24 16:55:39,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:39,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244ce0d613d77a40b8ac599528726c2857 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244ce0d613d77a40b8ac599528726c2857 2023-03-24 16:55:39,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:39,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=978, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/bd75234569784364864ab42080f5cf30 2023-03-24 16:55:40,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/2bb8be2fc9254e5bbdb3a2084565e435 2023-03-24 16:55:40,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/6a68f9a416b145198d42c6fd72bfc4da 2023-03-24 16:55:40,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/A/bd75234569784364864ab42080f5cf30 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bd75234569784364864ab42080f5cf30 2023-03-24 16:55:40,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bd75234569784364864ab42080f5cf30, entries=50, sequenceid=978, filesize=11.7 K 2023-03-24 16:55:40,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/B/2bb8be2fc9254e5bbdb3a2084565e435 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bb8be2fc9254e5bbdb3a2084565e435 2023-03-24 16:55:40,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bb8be2fc9254e5bbdb3a2084565e435, entries=50, sequenceid=978, filesize=7.1 K 2023-03-24 16:55:40,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/.tmp/C/6a68f9a416b145198d42c6fd72bfc4da as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/6a68f9a416b145198d42c6fd72bfc4da 2023-03-24 16:55:40,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/6a68f9a416b145198d42c6fd72bfc4da, entries=50, sequenceid=978, filesize=7.1 K 2023-03-24 16:55:40,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for 0496a4cb656870861b4baf5a724e8ad6 in 1648ms, sequenceid=978, compaction requested=true 2023-03-24 16:55:40,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/dd3a746bb5b14a4ca916f7ff9aa1ee08, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d8f70a44fe6e4237b3446f38865c89ea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d7fc907a3757429489887111983f9f8d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/8fb98ba0c43c4bda918fe809062fb403, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/128abfc4712749fd8f802f620efa0090, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bfe2729792724061baedf85181c88c64, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e] to archive 2023-03-24 16:55:40,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:40,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a07d1d7d2c2b4f15880d41d9e4669188 2023-03-24 16:55:40,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/3dd1bf87bdbc43c997bcd3243b07abbe 2023-03-24 16:55:40,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b1b970c2fb2143688a4a970775971cbd 2023-03-24 16:55:40,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/daff293f82a145189e7ca3698ce8878d 2023-03-24 16:55:40,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/dd3a746bb5b14a4ca916f7ff9aa1ee08 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/dd3a746bb5b14a4ca916f7ff9aa1ee08 2023-03-24 16:55:40,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/c77f24791c9246dba6fc06fc21d80af1 2023-03-24 16:55:40,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9ad3cf30f61746469a68d2becdce24db 2023-03-24 16:55:40,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d8f70a44fe6e4237b3446f38865c89ea to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d8f70a44fe6e4237b3446f38865c89ea 2023-03-24 16:55:40,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/7c20849f45bf48f6adb493914b4459fb 2023-03-24 16:55:40,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/447d2154d20541c582e76226371d977d 2023-03-24 16:55:40,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d7fc907a3757429489887111983f9f8d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/d7fc907a3757429489887111983f9f8d 2023-03-24 16:55:40,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/92f4129ee9894291a0374eebb5aaf53e 2023-03-24 16:55:40,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/a4d01872d7e241fea0d28470af8af341 2023-03-24 16:55:40,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/81325a8cd41e476386a744dc2e760e27 2023-03-24 16:55:40,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/61ee50b1655a4c81af6d3376e1090acf 2023-03-24 16:55:40,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/8fb98ba0c43c4bda918fe809062fb403 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/8fb98ba0c43c4bda918fe809062fb403 2023-03-24 16:55:40,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/1cc3ce13f91c4a68a5c16d413ac13b75 2023-03-24 16:55:40,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/b8deee58942f48eabc83ba427ad66d57 2023-03-24 16:55:40,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/128abfc4712749fd8f802f620efa0090 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/128abfc4712749fd8f802f620efa0090 2023-03-24 16:55:40,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e1170d02d8b2414889858ccc2cab2922 2023-03-24 16:55:40,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/faaf7bc86fe943d0a0e2950a56b96370 2023-03-24 16:55:40,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bfe2729792724061baedf85181c88c64 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bfe2729792724061baedf85181c88c64 2023-03-24 16:55:40,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f995b753576c4a0c9e0d50fa53d0f9ae 2023-03-24 16:55:40,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e14d01d20df44f5fab3cdb4c0057f9eb 2023-03-24 16:55:40,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/32678adce4f2477d8a6a42f0e405835e 2023-03-24 16:55:40,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3e6194d90f954f4cb93a2576bb7b4a9b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/9fdf4163f6944b999573618978129244, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ecd6d7805f1c4a2aaee42abca5830a57, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/85bf92f3a0334a349edf5af7119dae82, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3947179c7099462a8132b9128f6930db, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/197f3b536f80415d8f7b32e8151c6d4c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384] to archive 2023-03-24 16:55:40,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:40,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/87b8ab541944408daeebc779d1062336 2023-03-24 16:55:40,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5ee274cbc6b34ab8b63fd450282c268e 2023-03-24 16:55:40,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/8859eabdaf334bb680cf5f388db7cdc8 2023-03-24 16:55:40,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3e6194d90f954f4cb93a2576bb7b4a9b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3e6194d90f954f4cb93a2576bb7b4a9b 2023-03-24 16:55:40,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4c8b25f8f1e43f0982fa86ff47da848 2023-03-24 16:55:40,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/cbe51fa9d51a471884671514570e83f9 2023-03-24 16:55:40,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/9fdf4163f6944b999573618978129244 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/9fdf4163f6944b999573618978129244 2023-03-24 16:55:40,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/5d8ac146fa0045e38cadabada6699793 2023-03-24 16:55:40,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/61e255a718d9491aa6c19b6f93fd0800 2023-03-24 16:55:40,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ecd6d7805f1c4a2aaee42abca5830a57 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ecd6d7805f1c4a2aaee42abca5830a57 2023-03-24 16:55:40,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/b00674f7d13f4219a96aae6ddeea604d 2023-03-24 16:55:40,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3f47d84dd12143e8a91deb67085adb97 2023-03-24 16:55:40,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/07ea8e09e25f4e59a98eafa32ee94afa 2023-03-24 16:55:40,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bf364b9198645c894d07b7577178087 2023-03-24 16:55:40,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/85bf92f3a0334a349edf5af7119dae82 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/85bf92f3a0334a349edf5af7119dae82 2023-03-24 16:55:40,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/244cb6891858461cbb0c9c2e53fb6bd4 2023-03-24 16:55:40,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/e94a6ce72dc54c68a14802ad7904f5f7 2023-03-24 16:55:40,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3947179c7099462a8132b9128f6930db to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/3947179c7099462a8132b9128f6930db 2023-03-24 16:55:40,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/ea564a3a754141c5ba7c5a10c1b662b1 2023-03-24 16:55:40,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/608d9b1e34554372abfcc42122637060 2023-03-24 16:55:40,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/197f3b536f80415d8f7b32e8151c6d4c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/197f3b536f80415d8f7b32e8151c6d4c 2023-03-24 16:55:40,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/46d86eb3706f45029311cc589466b7e0 2023-03-24 16:55:40,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f4375de3829e4a8788ff8958e270ba0c 2023-03-24 16:55:40,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/f0bdfa3590c0460a8f70e4dc61a9b15f 2023-03-24 16:55:40,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/049bf41e1ce04f6194cc785ebb68a384 2023-03-24 16:55:40,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/bdeacac84eff4b949d412a7c6624a1bf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d3a16620986641a0a30b2ff3b4646923, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/a8f777372e614fa5bfb1b2b8e269a8fb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e132a7700f274c5a9256658abe584f15, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/073dc84f268f41d09b50df63ac4fea2d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/914461a4817a452d87bab0ac0f739854, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780] to archive 2023-03-24 16:55:40,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:55:40,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/75b138be735945c08f4a0d194b0754f9 2023-03-24 16:55:40,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/cd9d35d1b27b410d8517eed2fcd21c6e 2023-03-24 16:55:40,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/b7b748e4d6474f728f1a8a2590500966 2023-03-24 16:55:40,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/bdeacac84eff4b949d412a7c6624a1bf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/bdeacac84eff4b949d412a7c6624a1bf 2023-03-24 16:55:40,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de5d6a03ad75453dbe246c077ce63dd6 2023-03-24 16:55:40,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e20e3065dc644fa1830e98de79b07034 2023-03-24 16:55:40,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d3a16620986641a0a30b2ff3b4646923 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d3a16620986641a0a30b2ff3b4646923 2023-03-24 16:55:40,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ef5220093c114afbb5f87fbeb10f80cf 2023-03-24 16:55:40,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5008a5897a514edab37239debdaa7f22 2023-03-24 16:55:40,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/a8f777372e614fa5bfb1b2b8e269a8fb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/a8f777372e614fa5bfb1b2b8e269a8fb 2023-03-24 16:55:40,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/c8f952977ff2467e9e11eea89ec2bd05 2023-03-24 16:55:40,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2c97329015c046a8a0db6733d5e59d24 2023-03-24 16:55:40,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/5fd65494ee964d4db6841ba0a62efd52 2023-03-24 16:55:40,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/035c130c052d4dbbaccb20a6810dfe1e 2023-03-24 16:55:40,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e132a7700f274c5a9256658abe584f15 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/e132a7700f274c5a9256658abe584f15 2023-03-24 16:55:40,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/57dc55256d6f4153af990572921c3957 2023-03-24 16:55:40,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/dc68592d9fc942459a0c0053735f94a4 2023-03-24 16:55:40,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/073dc84f268f41d09b50df63ac4fea2d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/073dc84f268f41d09b50df63ac4fea2d 2023-03-24 16:55:40,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/062b99e1207b4fb3aadcd4d90b73aa84 2023-03-24 16:55:40,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/31deb712ea414997bdd78db6f859beec 2023-03-24 16:55:40,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/914461a4817a452d87bab0ac0f739854 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/914461a4817a452d87bab0ac0f739854 2023-03-24 16:55:40,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/4ba17c1ddf3f4ae1853a340bf0e575f3 2023-03-24 16:55:40,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/d5b2125f1f764ff2a350a321e22712ef 2023-03-24 16:55:40,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/2d51b08bda5f4056b58c8a78a189575b 2023-03-24 16:55:40,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/de9685ad6d8a434e9306f3a10ff61780 2023-03-24 16:55:40,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits/981.seqid, newMaxSeqId=981, maxSeqId=4 2023-03-24 16:55:40,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6. 2023-03-24 16:55:40,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0496a4cb656870861b4baf5a724e8ad6: 2023-03-24 16:55:40,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:40,850 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=0496a4cb656870861b4baf5a724e8ad6, regionState=CLOSED 2023-03-24 16:55:40,851 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676940850"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676940850"}]},"ts":"1679676940850"} 2023-03-24 16:55:40,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-24 16:55:40,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 0496a4cb656870861b4baf5a724e8ad6, server=jenkins-hbase4.apache.org,33315,1679676819242 in 3.3500 sec 2023-03-24 16:55:40,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-24 16:55:40,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0496a4cb656870861b4baf5a724e8ad6, UNASSIGN in 3.3540 sec 2023-03-24 16:55:40,854 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676940854"}]},"ts":"1679676940854"} 2023-03-24 16:55:40,855 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:55:40,858 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:55:40,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3670 sec 2023-03-24 16:55:40,986 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 16:55:41,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 16:55:41,601 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-24 16:55:41,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:55:41,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,603 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 16:55:41,604 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,607 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:41,609 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits] 2023-03-24 16:55:41,611 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/242de6a01b2a48b09646101274a8abcc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/242de6a01b2a48b09646101274a8abcc 2023-03-24 16:55:41,612 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9eb6628dd9f34ce7a2355ac60afdf8ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/9eb6628dd9f34ce7a2355ac60afdf8ab 2023-03-24 16:55:41,613 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bd75234569784364864ab42080f5cf30 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/bd75234569784364864ab42080f5cf30 2023-03-24 16:55:41,613 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e9fa7f2ddc2e4eee811f9b1392b18504 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/e9fa7f2ddc2e4eee811f9b1392b18504 2023-03-24 16:55:41,614 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f35b087cf5e84d098c4e5b82f8fcf1a2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/A/f35b087cf5e84d098c4e5b82f8fcf1a2 2023-03-24 16:55:41,616 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/1670f7523d454026885e0076ed2e0a31 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/1670f7523d454026885e0076ed2e0a31 2023-03-24 16:55:41,617 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bb8be2fc9254e5bbdb3a2084565e435 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/2bb8be2fc9254e5bbdb3a2084565e435 2023-03-24 16:55:41,618 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/44047f6a93e448d8805d325d222b3e3a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/44047f6a93e448d8805d325d222b3e3a 2023-03-24 16:55:41,619 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/59af51de83c041d584c71c9c4110fc4e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/59af51de83c041d584c71c9c4110fc4e 2023-03-24 16:55:41,620 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/c1237864d06d4eada59c35ac6251aeb3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/B/c1237864d06d4eada59c35ac6251aeb3 2023-03-24 16:55:41,621 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/004bd6310eeb44fd84362e3bef29323c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/004bd6310eeb44fd84362e3bef29323c 2023-03-24 16:55:41,622 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/3bfa2754758f4df8b8e50c1e44d422fb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/3bfa2754758f4df8b8e50c1e44d422fb 2023-03-24 16:55:41,623 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/6a68f9a416b145198d42c6fd72bfc4da to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/6a68f9a416b145198d42c6fd72bfc4da 2023-03-24 16:55:41,623 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/76529d5b3efd4802af26260b93ce82d2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/76529d5b3efd4802af26260b93ce82d2 2023-03-24 16:55:41,624 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ae702f8ed03448989b4515c33d508e37 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/C/ae702f8ed03448989b4515c33d508e37 2023-03-24 16:55:41,626 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits/981.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6/recovered.edits/981.seqid 2023-03-24 16:55:41,626 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/0496a4cb656870861b4baf5a724e8ad6 2023-03-24 16:55:41,626 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:55:41,627 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:55:41,627 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 16:55:41,629 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324067a87b3399245ee81a3e0861e5a8c72 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324067a87b3399245ee81a3e0861e5a8c72 2023-03-24 16:55:41,630 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240af0982689f04144b82a0161a1b99e94 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240af0982689f04144b82a0161a1b99e94 2023-03-24 16:55:41,631 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bcfd3b1478b43159234e5a56f344b15 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bcfd3b1478b43159234e5a56f344b15 2023-03-24 16:55:41,632 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241e08bc1b94f745c58b53f3d2b71f2287 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241e08bc1b94f745c58b53f3d2b71f2287 2023-03-24 16:55:41,632 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422c3dfa4ff204836a4b3dbe5333351a6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422c3dfa4ff204836a4b3dbe5333351a6 2023-03-24 16:55:41,633 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324260d79876a944c35b1880345b1f7d3c8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324260d79876a944c35b1880345b1f7d3c8 2023-03-24 16:55:41,634 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324291a42d9212d45eda14af97ab37326b1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324291a42d9212d45eda14af97ab37326b1 2023-03-24 16:55:41,635 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324369993de42c64d30a3bc97ead29725e9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324369993de42c64d30a3bc97ead29725e9 2023-03-24 16:55:41,636 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a7aa82c50f347b2b9859dadcf70a7ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a7aa82c50f347b2b9859dadcf70a7ab 2023-03-24 16:55:41,636 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324449b7cba8d7c4f208200845c1ae6275e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324449b7cba8d7c4f208200845c1ae6275e 2023-03-24 16:55:41,637 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244ce0d613d77a40b8ac599528726c2857 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244ce0d613d77a40b8ac599528726c2857 2023-03-24 16:55:41,638 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245318a0fc2c3149508a4b8852eb6b62f8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245318a0fc2c3149508a4b8852eb6b62f8 2023-03-24 16:55:41,639 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246561890b4fc545b194793cc8a98fb4eb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246561890b4fc545b194793cc8a98fb4eb 2023-03-24 16:55:41,640 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246dd40faea1244948b95b713c7945adfa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246dd40faea1244948b95b713c7945adfa 2023-03-24 16:55:41,641 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fd16624024146f1abe0e9a88e48b7f7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fd16624024146f1abe0e9a88e48b7f7 2023-03-24 16:55:41,641 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032497b42be37b3c4819a93382c91876f6d6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032497b42be37b3c4819a93382c91876f6d6 2023-03-24 16:55:41,642 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a9c23751bf88443c9b21852b12f05c0b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a9c23751bf88443c9b21852b12f05c0b 2023-03-24 16:55:41,643 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ad295cae74a947a2a5ec897e45f1c50b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ad295cae74a947a2a5ec897e45f1c50b 2023-03-24 16:55:41,644 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b88ced0b53e342d0a31c397846566fe6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b88ced0b53e342d0a31c397846566fe6 2023-03-24 16:55:41,645 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d142b53a7c8442f6a4bf111b68fce175 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d142b53a7c8442f6a4bf111b68fce175 2023-03-24 16:55:41,646 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d6af58a57e3440d1a97342392514e086 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d6af58a57e3440d1a97342392514e086 2023-03-24 16:55:41,646 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ea70413382e944a7a25dffa1a2546df4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ea70413382e944a7a25dffa1a2546df4 2023-03-24 16:55:41,647 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f0a3fac636d24fc98762283c4b621b7f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f0a3fac636d24fc98762283c4b621b7f 2023-03-24 16:55:41,648 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:55:41,649 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,650 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:55:41,652 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:55:41,652 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,652 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:55:41,653 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676941652"}]},"ts":"9223372036854775807"} 2023-03-24 16:55:41,654 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:55:41,654 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0496a4cb656870861b4baf5a724e8ad6, NAME => 'TestAcidGuarantees,,1679676906449.0496a4cb656870861b4baf5a724e8ad6.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:55:41,654 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:55:41,654 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676941654"}]},"ts":"9223372036854775807"} 2023-03-24 16:55:41,655 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:55:41,660 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-03-24 16:55:41,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 16:55:41,705 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-24 16:55:41,716 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 227), OpenFileDescriptor=508 (was 573), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 340), ProcessCount=169 (was 169), AvailableMemoryMB=5447 (was 5602) 2023-03-24 16:55:41,727 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=508, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=169, AvailableMemoryMB=5447 2023-03-24 16:55:41,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:55:41,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:55:41,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:55:41,732 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:55:41,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-24 16:55:41,732 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:55:41,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:41,733 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:41,734 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 empty. 2023-03-24 16:55:41,734 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:41,734 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:55:41,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:42,142 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:55:42,143 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 60e0f091d1c4c40e5cc075c89fc00205, NAME => 'TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:55:42,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:42,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:55:42,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 60e0f091d1c4c40e5cc075c89fc00205, disabling compactions & flushes 2023-03-24 16:55:42,549 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. after waiting 0 ms 2023-03-24 16:55:42,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,549 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,550 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:42,551 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:55:42,552 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676942552"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676942552"}]},"ts":"1679676942552"} 2023-03-24 16:55:42,553 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:55:42,553 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:55:42,554 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676942554"}]},"ts":"1679676942554"} 2023-03-24 16:55:42,554 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:55:42,557 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, ASSIGN}] 2023-03-24 16:55:42,558 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, ASSIGN 2023-03-24 16:55:42,559 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:55:42,710 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=60e0f091d1c4c40e5cc075c89fc00205, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:42,710 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676942710"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676942710"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676942710"}]},"ts":"1679676942710"} 2023-03-24 16:55:42,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:55:42,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:42,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 60e0f091d1c4c40e5cc075c89fc00205, NAME => 'TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:55:42,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:55:42,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,867 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,868 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A 2023-03-24 16:55:42,868 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A 2023-03-24 16:55:42,868 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:42,868 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60e0f091d1c4c40e5cc075c89fc00205 columnFamilyName A 2023-03-24 16:55:42,869 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(310): Store=60e0f091d1c4c40e5cc075c89fc00205/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:42,869 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,870 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B 2023-03-24 16:55:42,870 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B 2023-03-24 16:55:42,870 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:42,870 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60e0f091d1c4c40e5cc075c89fc00205 columnFamilyName B 2023-03-24 16:55:42,870 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(310): Store=60e0f091d1c4c40e5cc075c89fc00205/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:42,870 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,871 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C 2023-03-24 16:55:42,871 DEBUG [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C 2023-03-24 16:55:42,871 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:55:42,871 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60e0f091d1c4c40e5cc075c89fc00205 columnFamilyName C 2023-03-24 16:55:42,872 INFO [StoreOpener-60e0f091d1c4c40e5cc075c89fc00205-1] regionserver.HStore(310): Store=60e0f091d1c4c40e5cc075c89fc00205/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:55:42,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:55:42,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:42,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:55:42,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 60e0f091d1c4c40e5cc075c89fc00205; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68788685, jitterRate=0.025031283497810364}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:55:42,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:42,877 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., pid=49, masterSystemTime=1679676942862 2023-03-24 16:55:42,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:42,878 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=60e0f091d1c4c40e5cc075c89fc00205, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:42,878 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676942878"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676942878"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676942878"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676942878"}]},"ts":"1679676942878"} 2023-03-24 16:55:42,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-24 16:55:42,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 in 168 msec 2023-03-24 16:55:42,881 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-24 16:55:42,881 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, ASSIGN in 323 msec 2023-03-24 16:55:42,882 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:55:42,882 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676942882"}]},"ts":"1679676942882"} 2023-03-24 16:55:42,883 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:55:42,884 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:55:42,885 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-03-24 16:55:43,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 16:55:43,837 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-24 16:55:43,838 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x73eb7080 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,843 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@681750df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,844 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,846 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,847 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:55:43,849 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:55:43,850 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x2a987907 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,853 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2df63a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,854 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,857 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,858 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,862 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,862 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,867 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,867 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,872 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,873 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,876 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,876 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,880 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,880 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,885 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,885 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,892 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,892 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:55:43,895 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:55:43,899 DEBUG [hconnection-0x15bdbc52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,899 DEBUG [hconnection-0x6014bd2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:43,900 DEBUG [hconnection-0x4e02c2dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,901 DEBUG [hconnection-0x3b451444-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,901 DEBUG [hconnection-0x57609c58-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,901 DEBUG [hconnection-0x5a62e14d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,902 DEBUG [hconnection-0x136eafe0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,903 DEBUG [hconnection-0x1536998d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,903 DEBUG [hconnection-0x1948d252-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,903 DEBUG [hconnection-0x3a21a6ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:55:43,903 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51822, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,907 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51842, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,911 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51850, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,912 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51838, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:43,912 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51854, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,913 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:43,913 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:43,913 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:43,913 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:43,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51876, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,914 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51884, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51890, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:43,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:43,918 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:55:43,923 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:43,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:43,924 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:43,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:43,929 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,929 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:43,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:43,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:43,930 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:43,930 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:43,933 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:55:43,933 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:43,934 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:43,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677003949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677003952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:43,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677003976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677003976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:43,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677003976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677004077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677004077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677004084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677004084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677004084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677004281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677004281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677004287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677004288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677004288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,346 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/5feb780502bc4743acf57a3c004aef90 2023-03-24 16:55:44,370 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/83b1378ab79549dabc803313efcf3b70 2023-03-24 16:55:44,382 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/e43ed1c267dc4738be91d4ee73acbe7a 2023-03-24 16:55:44,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/5feb780502bc4743acf57a3c004aef90 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90 2023-03-24 16:55:44,393 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90, entries=200, sequenceid=15, filesize=13.9 K 2023-03-24 16:55:44,394 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/83b1378ab79549dabc803313efcf3b70 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70 2023-03-24 16:55:44,401 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 16:55:44,402 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/e43ed1c267dc4738be91d4ee73acbe7a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a 2023-03-24 16:55:44,406 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 16:55:44,407 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 60e0f091d1c4c40e5cc075c89fc00205 in 474ms, sequenceid=15, compaction requested=false 2023-03-24 16:55:44,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 16:55:44,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:44,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,410 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1348b285[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:44,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:44,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,413 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:44,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:44,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:44,413 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:44,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:44,417 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:44,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:44,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:44,417 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:44,417 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:44,419 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,419 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,419 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:44,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:44,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:44,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:44,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:44,424 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:44,424 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:44,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:44,524 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:44,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:44,528 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:44,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:44,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:44,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:44,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:44,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:44,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:44,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:44,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:44,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:44,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:44,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:44,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:44,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:44,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:44,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:44,531 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:44,532 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:44,532 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:44,539 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/aa3ecaf3ce784e0c9dd960c5615fff5b 2023-03-24 16:55:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:44,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:44,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677004596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677004598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677004598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677004599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677004599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677004704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677004704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677004704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677004704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677004909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677004909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677004910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:44,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677004910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,950 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/a9cefc7247c144978f55d6f239c00fcd 2023-03-24 16:55:44,968 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/37709b7d6c204385aa7f3b23e784759e 2023-03-24 16:55:44,972 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/aa3ecaf3ce784e0c9dd960c5615fff5b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b 2023-03-24 16:55:44,975 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b, entries=150, sequenceid=35, filesize=11.6 K 2023-03-24 16:55:44,975 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/a9cefc7247c144978f55d6f239c00fcd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd 2023-03-24 16:55:44,979 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd, entries=150, sequenceid=35, filesize=11.6 K 2023-03-24 16:55:44,979 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/37709b7d6c204385aa7f3b23e784759e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e 2023-03-24 16:55:44,982 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e, entries=150, sequenceid=35, filesize=11.6 K 2023-03-24 16:55:44,983 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 60e0f091d1c4c40e5cc075c89fc00205 in 451ms, sequenceid=35, compaction requested=false 2023-03-24 16:55:44,983 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:44,983 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:44,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,984 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52044d5f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:44,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:44,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:44,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,991 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:44,992 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:44,994 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:44,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:44,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:44,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:44,994 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:44,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:44,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:44,998 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:44,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:44,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,000 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,001 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,001 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,101 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:45,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:45,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,104 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,104 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,105 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,106 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,108 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,110 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,111 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,111 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/55678991d03f40e78cdedab908038dde 2023-03-24 16:55:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6579e21f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,112 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,112 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,115 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,120 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,120 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,126 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,126 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,127 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,127 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677005177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677005212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677005213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677005213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677005214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,227 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,230 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,233 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52bf883b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,241 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,246 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,246 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,255 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,255 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677005286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,355 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,358 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,358 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,360 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,360 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,360 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,363 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5d473[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,366 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,368 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,368 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,370 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,370 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,372 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,372 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,372 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,376 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,376 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,376 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,376 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,476 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,480 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,486 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,495 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,495 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,495 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,495 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677005491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,496 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742b0f0f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,499 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,502 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,508 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,512 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,512 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,513 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/78212be0d0a6455c86b937f00af127a9 2023-03-24 16:55:45,613 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,616 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,618 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,618 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,621 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee8628f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,625 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,625 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,626 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,629 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,629 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,632 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,635 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,635 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677005721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677005721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677005721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677005723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,735 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,738 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,740 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,742 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,742 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,744 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7038c739[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,747 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,747 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,748 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,751 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,751 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,756 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,758 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,758 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:45,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677005798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,858 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,865 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,866 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,866 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,867 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,868 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:45,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:45,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:45,870 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:45,871 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a849301[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:45,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:45,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:45,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:45,882 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:45,884 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:45,884 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:45,884 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:45,884 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:45,884 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:45,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:45,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:45,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:45,886 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,889 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,895 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:45,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:45,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:45,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:45,895 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:45,895 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:45,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:45,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:45,896 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,896 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:45,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/25fb0cd64405469d9fb6969f28aec9ce 2023-03-24 16:55:45,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/55678991d03f40e78cdedab908038dde as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde 2023-03-24 16:55:45,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde, entries=200, sequenceid=48, filesize=13.9 K 2023-03-24 16:55:45,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/78212be0d0a6455c86b937f00af127a9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9 2023-03-24 16:55:45,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9, entries=150, sequenceid=48, filesize=11.6 K 2023-03-24 16:55:45,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/25fb0cd64405469d9fb6969f28aec9ce as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce 2023-03-24 16:55:45,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce, entries=150, sequenceid=48, filesize=11.6 K 2023-03-24 16:55:45,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 60e0f091d1c4c40e5cc075c89fc00205 in 852ms, sequenceid=48, compaction requested=true 2023-03-24 16:55:45,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:45,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:45,956 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:45,956 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:45,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:45,957 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:45,957 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:45,957 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:45,957 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:45,957 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:45,957 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,957 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,957 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=39.4 K 2023-03-24 16:55:45,958 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=34.8 K 2023-03-24 16:55:45,958 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5feb780502bc4743acf57a3c004aef90, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679676943927 2023-03-24 16:55:45,958 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 83b1378ab79549dabc803313efcf3b70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679676943927 2023-03-24 16:55:45,958 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting aa3ecaf3ce784e0c9dd960c5615fff5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679676943951 2023-03-24 16:55:45,958 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a9cefc7247c144978f55d6f239c00fcd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679676943951 2023-03-24 16:55:45,958 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 55678991d03f40e78cdedab908038dde, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:45,959 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 78212be0d0a6455c86b937f00af127a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:45,964 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:45,965 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:45,973 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/4a0cfa68b97f444fb716907454d286c3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/4a0cfa68b97f444fb716907454d286c3 2023-03-24 16:55:45,975 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/ddf65a12e55343d9b45ee54da8082829 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/ddf65a12e55343d9b45ee54da8082829 2023-03-24 16:55:45,978 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 4a0cfa68b97f444fb716907454d286c3(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:45,978 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:45,978 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676945956; duration=0sec 2023-03-24 16:55:45,978 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:45,978 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:45,979 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into ddf65a12e55343d9b45ee54da8082829(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:45,979 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:45,979 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:45,979 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=13, startTime=1679676945956; duration=0sec 2023-03-24 16:55:45,979 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:45,979 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:45,979 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:45,979 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=34.8 K 2023-03-24 16:55:45,979 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e43ed1c267dc4738be91d4ee73acbe7a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679676943927 2023-03-24 16:55:45,980 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 37709b7d6c204385aa7f3b23e784759e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679676943951 2023-03-24 16:55:45,981 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 25fb0cd64405469d9fb6969f28aec9ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:45,985 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:45,992 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/4cd0001f5e044fe0ba48f0d7084581cb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/4cd0001f5e044fe0ba48f0d7084581cb 2023-03-24 16:55:45,996 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:45,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:45,997 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 4cd0001f5e044fe0ba48f0d7084581cb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:45,997 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:45,997 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676945957; duration=0sec 2023-03-24 16:55:45,997 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:45,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:45,999 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:45,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:45,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:45,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:45,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:45,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:46,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:46,003 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:46,003 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,009 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/d4a8b80475e041fb81dee936111d6bd0 2023-03-24 16:55:46,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:46,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:46,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677006345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,381 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:55:46,420 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/caddc839ad7b46a7b4f4462f3393c7db 2023-03-24 16:55:46,429 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/8efdd1e0fde04058b13108d47ca734a6 2023-03-24 16:55:46,433 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/d4a8b80475e041fb81dee936111d6bd0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0 2023-03-24 16:55:46,436 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 16:55:46,436 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/caddc839ad7b46a7b4f4462f3393c7db as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db 2023-03-24 16:55:46,439 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 16:55:46,440 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/8efdd1e0fde04058b13108d47ca734a6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6 2023-03-24 16:55:46,443 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 16:55:46,444 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 60e0f091d1c4c40e5cc075c89fc00205 in 441ms, sequenceid=70, compaction requested=false 2023-03-24 16:55:46,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:46,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:46,447 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,447 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,448 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767bac29[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:46,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:46,449 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,450 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:46,450 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,450 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,452 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:46,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:46,455 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:46,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:46,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:46,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:46,455 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:46,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:46,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:46,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:46,456 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,456 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,457 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,462 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,466 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:46,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:46,466 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,467 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:46,467 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:46,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,467 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677006510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,567 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:46,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:46,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:46,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:46,570 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:46,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:46,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:46,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,570 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:46,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:46,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:46,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9a4473[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:46,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:46,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,579 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:46,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:46,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:46,579 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:46,580 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:46,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:46,583 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:46,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:46,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:46,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:46,583 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:46,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:46,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:46,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:46,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,589 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:46,589 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677006617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,689 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:46,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:46,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:46,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:46,693 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:46,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:46,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:46,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,693 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,694 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:46,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:46,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:46,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:46,696 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:46,698 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d2d4736[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:46,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:46,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:46,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,701 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:46,704 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:46,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:46,706 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:46,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:46,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:46,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:46,706 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:46,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,707 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:46,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:46,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,713 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,714 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:46,714 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:46,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677006727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677006730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677006731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677006733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,814 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:46,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:46,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:46,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:46,817 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:46,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:46,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:46,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,819 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,819 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:46,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:46,821 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,821 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:46,821 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:46,821 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:46,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:46,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:46,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23cfdb09[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:46,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:46,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,826 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,826 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:46,826 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:46,827 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,827 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,828 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:46,829 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:46,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:46,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:46,830 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:46,830 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:46,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:46,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677006826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,836 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,838 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,839 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:46,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,839 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:46,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/d9a8888020e548468bc749d467354ab7 2023-03-24 16:55:46,939 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:46,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:46,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:46,942 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:46,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:46,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:46,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,943 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:46,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:46,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:46,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:46,946 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:46,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,948 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:46,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87e21bd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:46,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:46,949 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,951 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,951 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,953 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:46,955 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:46,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:46,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:46,955 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:46,955 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:46,956 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:46,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,957 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,957 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:46,958 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:46,960 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:46,960 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:46,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:46,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:46,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:46,960 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:46,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:46,961 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:46,961 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:46,961 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,065 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,068 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,072 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:47,078 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:47,080 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:47,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e551d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:47,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:47,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,083 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:47,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,084 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,085 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:47,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:47,088 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:47,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:47,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:47,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:47,088 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:47,091 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,091 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,091 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,095 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:47,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:47,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:47,095 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:47,095 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:47,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:47,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677007135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,195 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,199 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,202 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:47,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:47,212 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:47,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddb0f3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:47,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:47,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,215 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:47,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,215 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:47,215 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:47,217 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,217 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:47,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:47,219 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:47,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:47,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:47,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:47,219 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:47,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,222 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,225 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:47,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,226 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:47,226 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:47,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/ef998687799a4ec58deb3bc5a9aa3f17 2023-03-24 16:55:47,326 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,329 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,331 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,331 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:47,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:47,333 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:47,335 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9724b2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:47,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:47,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:47,341 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:47,344 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:47,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:47,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:47,344 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:47,344 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:47,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,346 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,346 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:47,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:47,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:47,350 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:47,350 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,350 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,450 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,453 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,455 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,457 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,457 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,457 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:47,457 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:47,460 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229d85b4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:47,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:47,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:47,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,463 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:47,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:47,467 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:47,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:47,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:47,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:47,467 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:47,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:47,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:47,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,473 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:47,473 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:47,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,573 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,576 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,576 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,578 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,580 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,580 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,580 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:47,580 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:47,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:47,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6188fa47[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:47,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:47,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:47,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,589 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:47,590 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:47,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:47,592 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:47,592 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:47,592 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:47,592 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:47,592 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:47,595 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,595 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,595 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:47,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:47,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,597 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:47,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:47,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:47,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:47,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:47,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,600 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:47,600 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:47,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:47,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:47,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:47,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677007641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/380cbe7367564960a3b4feb14df08869 2023-03-24 16:55:47,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/d9a8888020e548468bc749d467354ab7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7 2023-03-24 16:55:47,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7, entries=200, sequenceid=84, filesize=13.9 K 2023-03-24 16:55:47,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/ef998687799a4ec58deb3bc5a9aa3f17 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17 2023-03-24 16:55:47,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17, entries=150, sequenceid=84, filesize=11.6 K 2023-03-24 16:55:47,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/380cbe7367564960a3b4feb14df08869 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869 2023-03-24 16:55:47,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869, entries=150, sequenceid=84, filesize=11.6 K 2023-03-24 16:55:47,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 1243ms, sequenceid=84, compaction requested=true 2023-03-24 16:55:47,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:47,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:47,698 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:47,698 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:47,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:47,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:47,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:47,699 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:47,699 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:47,699 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:47,699 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,699 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,699 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/ddf65a12e55343d9b45ee54da8082829, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=37.2 K 2023-03-24 16:55:47,699 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/4a0cfa68b97f444fb716907454d286c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=34.9 K 2023-03-24 16:55:47,700 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ddf65a12e55343d9b45ee54da8082829, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:47,700 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4a0cfa68b97f444fb716907454d286c3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:47,700 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:47,700 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d4a8b80475e041fb81dee936111d6bd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679676945164 2023-03-24 16:55:47,700 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting caddc839ad7b46a7b4f4462f3393c7db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679676945164 2023-03-24 16:55:47,701 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d9a8888020e548468bc749d467354ab7, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946317 2023-03-24 16:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:47,701 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ef998687799a4ec58deb3bc5a9aa3f17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946338 2023-03-24 16:55:47,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:47,703 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:47,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:47,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:47,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:47,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:47,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:47,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:47,707 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:47,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:47,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:47,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:47,712 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,712 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:47,712 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:47,712 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:47,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:47,727 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/f00a68ef715441e8aaa4a95f0ee50e0a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f00a68ef715441e8aaa4a95f0ee50e0a 2023-03-24 16:55:47,730 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into f00a68ef715441e8aaa4a95f0ee50e0a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:47,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:47,730 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676947698; duration=0sec 2023-03-24 16:55:47,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:47,730 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:47,731 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:47,731 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:47,732 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:47,732 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/4cd0001f5e044fe0ba48f0d7084581cb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=34.9 K 2023-03-24 16:55:47,732 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4cd0001f5e044fe0ba48f0d7084581cb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679676944597 2023-03-24 16:55:47,732 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8efdd1e0fde04058b13108d47ca734a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679676945164 2023-03-24 16:55:47,733 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 380cbe7367564960a3b4feb14df08869, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946338 2023-03-24 16:55:47,740 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:47,761 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/11552c00b1af45bc945ba65c11e32249 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/11552c00b1af45bc945ba65c11e32249 2023-03-24 16:55:47,765 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 11552c00b1af45bc945ba65c11e32249(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:47,765 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:47,765 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676947698; duration=0sec 2023-03-24 16:55:47,765 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:48,142 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/02e14a197b0a48939935e1063de4a16f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/02e14a197b0a48939935e1063de4a16f 2023-03-24 16:55:48,146 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into 02e14a197b0a48939935e1063de4a16f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:48,146 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:48,146 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=13, startTime=1679676947698; duration=0sec 2023-03-24 16:55:48,146 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:48,149 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/f352ab83befe4a0c9878ce2282ea59e0 2023-03-24 16:55:48,558 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/a52d5007ddc342dfb6d046b1bfda470c 2023-03-24 16:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:48,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:48,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677008694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677008738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,746 DEBUG [Thread-3247] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:48,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677008746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,753 DEBUG [Thread-3251] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677008754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677008754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,761 DEBUG [Thread-3249] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:48,763 DEBUG [Thread-3253] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:55:48,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677008798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:48,968 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/013a07e884d14065b33155a999578eb0 2023-03-24 16:55:48,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/f352ab83befe4a0c9878ce2282ea59e0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0 2023-03-24 16:55:48,982 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0, entries=150, sequenceid=102, filesize=11.6 K 2023-03-24 16:55:48,983 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/a52d5007ddc342dfb6d046b1bfda470c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c 2023-03-24 16:55:48,989 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c, entries=150, sequenceid=102, filesize=11.6 K 2023-03-24 16:55:48,990 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/013a07e884d14065b33155a999578eb0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0 2023-03-24 16:55:48,998 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0, entries=150, sequenceid=102, filesize=11.6 K 2023-03-24 16:55:48,999 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 1287ms, sequenceid=102, compaction requested=false 2023-03-24 16:55:48,999 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:48,999 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:49,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab984ad[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:49,008 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:49,008 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,009 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:49,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:49,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:49,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 16:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:49,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:49,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:49,016 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:49,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:49,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:49,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:49,016 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:49,019 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,019 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,019 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:49,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,026 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,026 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,026 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:49,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/c808b5238b6743d0acdb8f236650e627 2023-03-24 16:55:49,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677009097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,126 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:49,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:49,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:49,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:49,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:49,129 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:49,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:49,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:49,131 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,131 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:49,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:49,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,136 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686983fc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:49,139 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:49,139 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:49,142 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,142 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:49,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,143 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,143 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,144 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:49,146 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:49,146 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:49,146 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:49,146 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:49,146 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:49,147 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,147 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,147 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,152 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:49,153 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:49,153 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677009206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,253 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:49,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:49,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:49,256 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:49,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,264 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,264 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,265 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,265 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:49,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:49,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,270 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26971b60[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:49,272 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:49,272 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,273 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:49,273 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:49,278 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:49,280 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:49,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:49,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:49,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:49,281 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:49,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,282 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,288 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,288 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:49,288 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,289 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:49,289 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:49,289 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,389 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:49,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:49,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:49,393 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:49,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:49,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:49,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,394 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:49,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:49,398 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,398 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:49,398 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:49,398 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:49,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8937afc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:49,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:49,403 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:49,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:49,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:49,407 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:49,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:49,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:49,407 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:49,407 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:49,408 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:49,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,415 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,415 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677009413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8717b88f6e4a4158ada6e58422da2507 2023-03-24 16:55:49,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/b4a5bc2f7f734f87b262109cbfe1f4d9 2023-03-24 16:55:49,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/c808b5238b6743d0acdb8f236650e627 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627 2023-03-24 16:55:49,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627, entries=200, sequenceid=120, filesize=13.9 K 2023-03-24 16:55:49,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8717b88f6e4a4158ada6e58422da2507 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507 2023-03-24 16:55:49,515 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:49,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:49,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:49,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507, entries=150, sequenceid=120, filesize=11.6 K 2023-03-24 16:55:49,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/b4a5bc2f7f734f87b262109cbfe1f4d9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9 2023-03-24 16:55:49,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:49,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:49,524 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:49,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:49,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:49,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:49,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:49,530 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,530 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:49,530 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:49,530 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:49,532 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eb8737[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:49,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:49,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,537 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9, entries=150, sequenceid=120, filesize=11.6 K 2023-03-24 16:55:49,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:49,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:49,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:49,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 60e0f091d1c4c40e5cc075c89fc00205 in 525ms, sequenceid=120, compaction requested=true 2023-03-24 16:55:49,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:49,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:49,538 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:49,538 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:49,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:49,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:49,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:49,539 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:49,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:49,539 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:49,539 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,539 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,539 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/02e14a197b0a48939935e1063de4a16f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=37.3 K 2023-03-24 16:55:49,539 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f00a68ef715441e8aaa4a95f0ee50e0a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=35.0 K 2023-03-24 16:55:49,545 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f00a68ef715441e8aaa4a95f0ee50e0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946338 2023-03-24 16:55:49,545 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 02e14a197b0a48939935e1063de4a16f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946338 2023-03-24 16:55:49,545 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a52d5007ddc342dfb6d046b1bfda470c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679676946496 2023-03-24 16:55:49,545 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f352ab83befe4a0c9878ce2282ea59e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679676946496 2023-03-24 16:55:49,546 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8717b88f6e4a4158ada6e58422da2507, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948685 2023-03-24 16:55:49,547 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c808b5238b6743d0acdb8f236650e627, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948652 2023-03-24 16:55:49,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:49,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:49,555 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:49,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:49,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:49,555 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:49,555 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:49,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:49,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:49,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:49,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,568 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#434 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:49,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:49,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:49,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:49,569 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:49,569 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:49,577 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#435 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:49,620 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/0816675fb4df4e5ca0d20be035c93631 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0816675fb4df4e5ca0d20be035c93631 2023-03-24 16:55:49,625 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into 0816675fb4df4e5ca0d20be035c93631(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:49,625 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:49,625 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=13, startTime=1679676949538; duration=0sec 2023-03-24 16:55:49,625 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:49,625 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:49,626 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:49,627 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:49,627 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,627 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/11552c00b1af45bc945ba65c11e32249, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=35.0 K 2023-03-24 16:55:49,627 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 11552c00b1af45bc945ba65c11e32249, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679676946338 2023-03-24 16:55:49,627 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 013a07e884d14065b33155a999578eb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679676946496 2023-03-24 16:55:49,628 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b4a5bc2f7f734f87b262109cbfe1f4d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948685 2023-03-24 16:55:49,637 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:49,669 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:49,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:49,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:49,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:49,672 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:49,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:49,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:49,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:49,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:49,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:49,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:49,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:49,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:49,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:49,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:49,676 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:49,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,677 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:49,677 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:49,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:49,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:49,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51850 deadline: 1679677009775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:49,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677009881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,021 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/1fdad176a491427c926afc053a1d4e1c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/1fdad176a491427c926afc053a1d4e1c 2023-03-24 16:55:50,025 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 1fdad176a491427c926afc053a1d4e1c(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:50,025 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:50,025 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676949538; duration=0sec 2023-03-24 16:55:50,025 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:50,046 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/3290e76a514b4d88b4abaa22d1130271 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3290e76a514b4d88b4abaa22d1130271 2023-03-24 16:55:50,050 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 3290e76a514b4d88b4abaa22d1130271(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:50,050 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:50,050 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676949538; duration=0sec 2023-03-24 16:55:50,050 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:50,086 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/95459c36f1da49c0ae7f8f47039a6b80 2023-03-24 16:55:50,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:50,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677010087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,102 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/70245985054a46f9918248a6e40f76c3 2023-03-24 16:55:50,113 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/9e42f73d872549edb1a756b89b55994d 2023-03-24 16:55:50,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/95459c36f1da49c0ae7f8f47039a6b80 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80 2023-03-24 16:55:50,131 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80, entries=150, sequenceid=138, filesize=11.8 K 2023-03-24 16:55:50,132 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/70245985054a46f9918248a6e40f76c3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3 2023-03-24 16:55:50,136 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3, entries=150, sequenceid=138, filesize=11.8 K 2023-03-24 16:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/9e42f73d872549edb1a756b89b55994d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d 2023-03-24 16:55:50,140 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d, entries=150, sequenceid=138, filesize=11.8 K 2023-03-24 16:55:50,141 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 60e0f091d1c4c40e5cc075c89fc00205 in 465ms, sequenceid=138, compaction requested=false 2023-03-24 16:55:50,141 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:50,141 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:50,144 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b1f76b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:50,149 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:50,151 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:50,155 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:50,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:50,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:50,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:50,155 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:50,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:50,161 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:50,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:50,170 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:50,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:50,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:50,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:50,170 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:50,170 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:50,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:50,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:50,271 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:50,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:50,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:50,275 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:50,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:50,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:50,275 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:50,277 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:50,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:50,280 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:50,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:50,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:50,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:50,285 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:50,286 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:50,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:50,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:50,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677010458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677010562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:50,707 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/22101baaa0824175a8f4edde8cb7884c 2023-03-24 16:55:50,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:50,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677010768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:51,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677011073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,117 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/b3fa0927c1104ecd8fd23303cf991a1d 2023-03-24 16:55:51,533 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/63085aa6bdbb436ba1967bda6a299503 2023-03-24 16:55:51,536 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/22101baaa0824175a8f4edde8cb7884c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c 2023-03-24 16:55:51,540 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c, entries=150, sequenceid=155, filesize=11.8 K 2023-03-24 16:55:51,540 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/b3fa0927c1104ecd8fd23303cf991a1d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d 2023-03-24 16:55:51,543 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d, entries=150, sequenceid=155, filesize=11.8 K 2023-03-24 16:55:51,543 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/63085aa6bdbb436ba1967bda6a299503 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503 2023-03-24 16:55:51,546 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503, entries=150, sequenceid=155, filesize=11.8 K 2023-03-24 16:55:51,547 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 60e0f091d1c4c40e5cc075c89fc00205 in 1262ms, sequenceid=155, compaction requested=true 2023-03-24 16:55:51,547 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:51,547 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62e3fa22[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:51,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:51,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,553 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:51,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:51,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:51,557 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:51,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:51,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:51,557 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:51,557 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:51,559 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,559 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:51,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,564 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,565 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,565 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:51,565 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:51,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:51,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:51,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:51,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:51,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:51,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/0a3cd7035922453685f373fa4f2c068b 2023-03-24 16:55:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677011650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,665 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:51,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:51,668 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:51,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:51,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:51,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:51,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:51,671 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,671 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:51,671 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:51,671 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c31b105[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:51,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:51,675 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:51,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,679 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:51,681 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:51,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:51,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:51,681 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:51,681 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:51,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,684 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,687 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:51,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,688 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:51,688 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,688 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,688 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677011752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,788 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:51,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:51,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:51,791 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:51,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:51,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:51,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,791 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,793 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,793 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,793 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,793 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:51,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:51,794 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,794 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:51,794 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:51,795 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,797 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20bf538b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:51,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:51,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,800 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:51,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:51,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:51,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:51,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:51,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:51,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:51,806 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:51,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,810 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:51,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,816 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:51,816 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:51,916 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:51,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:51,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:51,919 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:51,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:51,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:51,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,920 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,921 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:51,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:51,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:51,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:51,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:51,925 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:51,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fcf258[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:51,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:51,926 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,934 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,934 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:51,935 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,935 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:51,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:51,939 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:51,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:51,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:51,940 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:51,940 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:51,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,941 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,942 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:51,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:51,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:51,947 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:51,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:51,947 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:51,948 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:51,947 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,948 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:51,948 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:51,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677011956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/0beee4141b7f41449ec8521bbb19270c 2023-03-24 16:55:52,048 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:52,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:52,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:52,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:52,051 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:52,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:52,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:52,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,051 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:52,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:52,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:52,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:52,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:52,055 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:52,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143327af[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:52,059 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:52,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:52,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:52,062 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:52,065 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:52,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:52,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:52,065 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:52,065 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:52,066 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,066 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,066 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,068 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,071 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:52,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,071 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,072 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:52,072 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:52,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,072 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,172 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:52,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:52,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:52,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:52,174 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:52,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:52,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:52,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,176 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:52,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:52,178 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,179 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:52,179 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:52,179 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@657b15b5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:52,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:52,183 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:52,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,185 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:52,188 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:52,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:52,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:52,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:52,189 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:52,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,190 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,192 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,194 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:52,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,194 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:52,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,195 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,195 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:52,195 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:52,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677012261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,295 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:52,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:52,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:52,299 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:52,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:52,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:52,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,308 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,308 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:52,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:52,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:52,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:52,311 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:52,315 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f82319a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:52,318 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:52,318 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,320 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:52,320 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:52,320 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,320 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:52,320 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:52,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:52,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:52,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:52,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:52,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:52,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:52,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:52,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:52,338 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:52,340 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:52,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:52,343 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:52,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:52,344 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:52,344 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:52,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:52,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:52,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/591d49d64b194bec8b7bd54859c7cfbb 2023-03-24 16:55:52,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/0a3cd7035922453685f373fa4f2c068b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b 2023-03-24 16:55:52,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b, entries=200, sequenceid=173, filesize=14.1 K 2023-03-24 16:55:52,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/0beee4141b7f41449ec8521bbb19270c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c 2023-03-24 16:55:52,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c, entries=150, sequenceid=173, filesize=11.8 K 2023-03-24 16:55:52,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/591d49d64b194bec8b7bd54859c7cfbb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb 2023-03-24 16:55:52,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb, entries=150, sequenceid=173, filesize=11.8 K 2023-03-24 16:55:52,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 849ms, sequenceid=173, compaction requested=true 2023-03-24 16:55:52,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:52,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:52,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:55:52,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 16:55:52,430 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:52,430 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:52,432 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:52,432 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:52,432 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,432 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0816675fb4df4e5ca0d20be035c93631, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=49.5 K 2023-03-24 16:55:52,432 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48302 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:52,432 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:52,432 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,433 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3290e76a514b4d88b4abaa22d1130271, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=47.2 K 2023-03-24 16:55:52,433 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0816675fb4df4e5ca0d20be035c93631, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948685 2023-03-24 16:55:52,433 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3290e76a514b4d88b4abaa22d1130271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948685 2023-03-24 16:55:52,433 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 95459c36f1da49c0ae7f8f47039a6b80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679676949078 2023-03-24 16:55:52,433 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9e42f73d872549edb1a756b89b55994d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679676949078 2023-03-24 16:55:52,433 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 22101baaa0824175a8f4edde8cb7884c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679676949755 2023-03-24 16:55:52,434 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 63085aa6bdbb436ba1967bda6a299503, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679676949755 2023-03-24 16:55:52,434 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0a3cd7035922453685f373fa4f2c068b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:52,434 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 591d49d64b194bec8b7bd54859c7cfbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:52,443 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:52,444 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:52,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:52,445 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:52,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:52,481 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:52,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:52,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:52,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,481 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,483 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:52,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:52,483 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:52,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:52,485 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/a31650bdd940446aa25e7f97b2d15624 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a31650bdd940446aa25e7f97b2d15624 2023-03-24 16:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:52,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:52,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:52,487 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,487 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:52,487 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:52,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:52,489 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into a31650bdd940446aa25e7f97b2d15624(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:52,489 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:52,489 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=12, startTime=1679676952430; duration=0sec 2023-03-24 16:55:52,489 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:52,489 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:52,491 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48302 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:52,491 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:52,491 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:52,491 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/1fdad176a491427c926afc053a1d4e1c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=47.2 K 2023-03-24 16:55:52,492 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1fdad176a491427c926afc053a1d4e1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679676948685 2023-03-24 16:55:52,492 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 70245985054a46f9918248a6e40f76c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679676949078 2023-03-24 16:55:52,494 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b3fa0927c1104ecd8fd23303cf991a1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679676949755 2023-03-24 16:55:52,495 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 0beee4141b7f41449ec8521bbb19270c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:52,495 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/d37a45deb8a04e3aa0008bbb4977ae30 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d37a45deb8a04e3aa0008bbb4977ae30 2023-03-24 16:55:52,499 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into d37a45deb8a04e3aa0008bbb4977ae30(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:52,500 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:52,500 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=12, startTime=1679676952430; duration=0sec 2023-03-24 16:55:52,500 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:52,502 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:52,514 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/14c02e971749404bb0c84bc14e96dfbc 2023-03-24 16:55:52,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:52,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:52,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677012786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677012787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677012788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677012789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677012791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677012892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677012892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677012895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677012895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:52,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677012897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:52,918 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8eba811ebe6c422290519822edb3a780 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8eba811ebe6c422290519822edb3a780 2023-03-24 16:55:52,922 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 8eba811ebe6c422290519822edb3a780(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:52,922 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:52,922 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=12, startTime=1679676952430; duration=0sec 2023-03-24 16:55:52,922 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:52,928 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/c211aa53dfb34a9bbb6b5f84cf424867 2023-03-24 16:55:53,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677013096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677013098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677013099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677013100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677013103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,347 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/de53b808f4fa42ceb8bc5876db496975 2023-03-24 16:55:53,354 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/14c02e971749404bb0c84bc14e96dfbc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc 2023-03-24 16:55:53,358 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc, entries=150, sequenceid=187, filesize=11.8 K 2023-03-24 16:55:53,361 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/c211aa53dfb34a9bbb6b5f84cf424867 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867 2023-03-24 16:55:53,380 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867, entries=150, sequenceid=187, filesize=11.8 K 2023-03-24 16:55:53,381 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/de53b808f4fa42ceb8bc5876db496975 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975 2023-03-24 16:55:53,389 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975, entries=150, sequenceid=187, filesize=11.8 K 2023-03-24 16:55:53,389 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 902ms, sequenceid=187, compaction requested=false 2023-03-24 16:55:53,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:53,390 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:53,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:53,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:53,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,391 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31627a00[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:53,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:53,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,395 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:53,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,396 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:53,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:53,405 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:53,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:53,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:53,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:53,405 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:53,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,409 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:53,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:53,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,416 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,416 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:53,417 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677013414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677013421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677013422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677013422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677013423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,517 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:53,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:53,520 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:53,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:53,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:53,526 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677013523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c612ddc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:53,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:53,528 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:53,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:53,532 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677013528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677013529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677013529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677013529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:53,534 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:53,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:53,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:53,535 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:53,535 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:53,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,538 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:53,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:53,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,541 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:53,542 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:53,542 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,542 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,542 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:53,642 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:53,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:53,645 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:53,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:53,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:53,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,648 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:53,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:53,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:53,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:53,652 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:53,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:53,653 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:53,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:53,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:53,653 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:53,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,655 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:53,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a4158[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:53,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:53,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,663 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:53,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,664 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:53,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:53,667 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:53,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:53,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:53,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:53,668 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:53,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,669 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,671 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,671 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:53,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:53,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:53,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,674 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:53,675 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,674 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,675 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677013729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677013734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677013734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677013735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:53,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677013736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,775 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:53,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:53,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:53,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:53,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:53,777 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:53,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:53,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:53,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:53,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:53,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:53,790 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:53,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2faabe02[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:53,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:53,794 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,795 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:53,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,798 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:53,799 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:53,801 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:53,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:53,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:53,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:53,801 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:53,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,805 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:53,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,810 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,812 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:53,813 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,813 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:53,813 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:53,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/6a5b91802adc41ff90a8369fec467b2e 2023-03-24 16:55:53,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/359cc6f8a3b344d48ce58df3a1501685 2023-03-24 16:55:53,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/eeabc83c9f1a4ce1bc92b5063d1b819d 2023-03-24 16:55:53,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/6a5b91802adc41ff90a8369fec467b2e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e 2023-03-24 16:55:53,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e, entries=200, sequenceid=209, filesize=14.1 K 2023-03-24 16:55:53,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/359cc6f8a3b344d48ce58df3a1501685 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685 2023-03-24 16:55:53,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685, entries=150, sequenceid=209, filesize=11.8 K 2023-03-24 16:55:53,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/eeabc83c9f1a4ce1bc92b5063d1b819d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d 2023-03-24 16:55:53,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d, entries=150, sequenceid=209, filesize=11.8 K 2023-03-24 16:55:53,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 466ms, sequenceid=209, compaction requested=true 2023-03-24 16:55:53,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:53,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:53,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:53,869 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:53,869 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:53,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:53,869 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38942 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:53,870 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,870 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,870 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8eba811ebe6c422290519822edb3a780, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=35.7 K 2023-03-24 16:55:53,870 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a31650bdd940446aa25e7f97b2d15624, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=38.0 K 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8eba811ebe6c422290519822edb3a780, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a31650bdd940446aa25e7f97b2d15624, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting c211aa53dfb34a9bbb6b5f84cf424867, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679676951620 2023-03-24 16:55:53,870 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 14c02e971749404bb0c84bc14e96dfbc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679676951620 2023-03-24 16:55:53,871 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 359cc6f8a3b344d48ce58df3a1501685, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952788 2023-03-24 16:55:53,871 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6a5b91802adc41ff90a8369fec467b2e, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952785 2023-03-24 16:55:53,877 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:53,881 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:53,895 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/3dcf8a6283d0442fa296b420d1947e63 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3dcf8a6283d0442fa296b420d1947e63 2023-03-24 16:55:53,896 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/bd485102054a433d99a204d4ddc7e3dc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bd485102054a433d99a204d4ddc7e3dc 2023-03-24 16:55:53,899 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 3dcf8a6283d0442fa296b420d1947e63(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:53,899 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:53,899 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676953869; duration=0sec 2023-03-24 16:55:53,899 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:53,899 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:55:53,900 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:55:53,900 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:53,900 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,900 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d37a45deb8a04e3aa0008bbb4977ae30, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=35.7 K 2023-03-24 16:55:53,901 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d37a45deb8a04e3aa0008bbb4977ae30, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1679676950446 2023-03-24 16:55:53,901 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting de53b808f4fa42ceb8bc5876db496975, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679676951620 2023-03-24 16:55:53,901 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting eeabc83c9f1a4ce1bc92b5063d1b819d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952788 2023-03-24 16:55:53,902 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into bd485102054a433d99a204d4ddc7e3dc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:53,902 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:53,902 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=13, startTime=1679676953869; duration=0sec 2023-03-24 16:55:53,902 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:53,906 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:53,913 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:53,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:53,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:53,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:53,922 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:53,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:53,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:53,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,924 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:53,925 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:53,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:53,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:53,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:53,927 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:53,927 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:53,927 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:53,945 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/bb88a0570a6a414a8f3969d0df2f3fdc 2023-03-24 16:55:53,955 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/26573d67ab304501acb88576baa11ead 2023-03-24 16:55:54,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:54,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:54,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677014053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677014056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677014056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677014058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677014058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677014161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677014168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677014168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677014168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677014168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,317 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/34e6dd9cff7a4f54a016b6a0ee023c1e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/34e6dd9cff7a4f54a016b6a0ee023c1e 2023-03-24 16:55:54,324 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 34e6dd9cff7a4f54a016b6a0ee023c1e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:54,324 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:54,324 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676953869; duration=0sec 2023-03-24 16:55:54,324 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:54,363 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/3ce614a6cfdb4c328a95d247eda1c29e 2023-03-24 16:55:54,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/bb88a0570a6a414a8f3969d0df2f3fdc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc 2023-03-24 16:55:54,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677014371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,376 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc, entries=100, sequenceid=225, filesize=9.4 K 2023-03-24 16:55:54,377 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/26573d67ab304501acb88576baa11ead as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead 2023-03-24 16:55:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677014374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677014375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677014375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677014375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,390 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead, entries=100, sequenceid=225, filesize=9.4 K 2023-03-24 16:55:54,390 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/3ce614a6cfdb4c328a95d247eda1c29e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e 2023-03-24 16:55:54,396 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e, entries=100, sequenceid=225, filesize=9.4 K 2023-03-24 16:55:54,397 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 470ms, sequenceid=225, compaction requested=false 2023-03-24 16:55:54,397 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:54,397 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,407 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:54,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,408 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3a738d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:54,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:54,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:54,414 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:54,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:54,414 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:54,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:54,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:54,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,421 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:54,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:54,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:54,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:54,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:54,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:54,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:54,426 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:54,426 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:54,527 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:54,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:54,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:54,544 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:54,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:54,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:54,544 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:54,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:54,546 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:54,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:54,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:54,546 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:54,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:54,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:54,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:54,561 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:54,561 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:54,562 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:54,562 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:54,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:54,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:54,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677014709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677014712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677014712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677014713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677014713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677014823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677014823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677014824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677014825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:54,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677014825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,000 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/a67e4722ba69401f96d0da2d41efd7da 2023-03-24 16:55:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677015032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,057 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/0c0b4131a5624b8689b0551cdcd6e7fc 2023-03-24 16:55:55,103 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/5d942e0184aa44ebb2a4b8dd5a2b00c9 2023-03-24 16:55:55,109 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/a67e4722ba69401f96d0da2d41efd7da as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da 2023-03-24 16:55:55,114 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da, entries=150, sequenceid=244, filesize=11.8 K 2023-03-24 16:55:55,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/0c0b4131a5624b8689b0551cdcd6e7fc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc 2023-03-24 16:55:55,139 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc, entries=150, sequenceid=244, filesize=11.8 K 2023-03-24 16:55:55,139 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/5d942e0184aa44ebb2a4b8dd5a2b00c9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9 2023-03-24 16:55:55,154 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9, entries=150, sequenceid=244, filesize=11.8 K 2023-03-24 16:55:55,155 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 594ms, sequenceid=244, compaction requested=true 2023-03-24 16:55:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:55,157 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,157 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3800b03b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:55,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:55,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:55,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:55,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:55,172 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:55,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:55,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:55,173 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:55,173 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:55,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:55,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,182 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:55,182 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:55,282 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:55,286 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:55,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:55,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:55,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:55,288 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:55,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:55,292 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,292 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:55,292 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:55,292 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,323 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/b1abf2a8252e424a8f262e55ed8086d3 2023-03-24 16:55:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:55,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:55,358 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/852c5d299b114d0092f76b3c53172fd6 2023-03-24 16:55:55,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,389 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/623b86cc28bc4b8884fd0b94257c4acb 2023-03-24 16:55:55,395 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/b1abf2a8252e424a8f262e55ed8086d3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3 2023-03-24 16:55:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677015386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,398 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3, entries=150, sequenceid=258, filesize=11.8 K 2023-03-24 16:55:55,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/852c5d299b114d0092f76b3c53172fd6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6 2023-03-24 16:55:55,414 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6, entries=150, sequenceid=258, filesize=11.8 K 2023-03-24 16:55:55,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/623b86cc28bc4b8884fd0b94257c4acb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb 2023-03-24 16:55:55,424 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb, entries=150, sequenceid=258, filesize=11.8 K 2023-03-24 16:55:55,425 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 133ms, sequenceid=258, compaction requested=true 2023-03-24 16:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:55,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bd336e2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:55,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:55,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:55,433 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:55,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:55,433 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:55,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,437 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,442 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:55,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:55,443 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:55,443 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:55,494 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677015511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/cc21c19ad33945798b022ad768f33619 2023-03-24 16:55:55,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,543 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:55,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:55,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8f560d47a35148bfbfe971bbe5c2d695 2023-03-24 16:55:55,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:55,552 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:55,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:55,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:55,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:55,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:55,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:55,568 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,568 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:55,568 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:55,568 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:55,570 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,570 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f3df34[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:55,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:55,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,573 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:55,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,574 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:55,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:55,580 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:55,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:55,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:55,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:55,580 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:55,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/ffa04b1359c14da296bbfbf5f2af59dc 2023-03-24 16:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/cc21c19ad33945798b022ad768f33619 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619 2023-03-24 16:55:55,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:55,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:55,591 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,592 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:55,592 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,592 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619, entries=200, sequenceid=277, filesize=14.3 K 2023-03-24 16:55:55,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8f560d47a35148bfbfe971bbe5c2d695 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695 2023-03-24 16:55:55,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695, entries=150, sequenceid=277, filesize=11.9 K 2023-03-24 16:55:55,603 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/ffa04b1359c14da296bbfbf5f2af59dc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc 2023-03-24 16:55:55,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc, entries=150, sequenceid=277, filesize=11.9 K 2023-03-24 16:55:55,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 60e0f091d1c4c40e5cc075c89fc00205 in 113ms, sequenceid=277, compaction requested=true 2023-03-24 16:55:55,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:55,607 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:55,607 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:55,609 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60972 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:55,609 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:55,609 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,609 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bd485102054a433d99a204d4ddc7e3dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=59.5 K 2023-03-24 16:55:55,609 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58532 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:55,609 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:55,610 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,610 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3dcf8a6283d0442fa296b420d1947e63, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=57.2 K 2023-03-24 16:55:55,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting bd485102054a433d99a204d4ddc7e3dc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952788 2023-03-24 16:55:55,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting bb88a0570a6a414a8f3969d0df2f3fdc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1679676953420 2023-03-24 16:55:55,610 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3dcf8a6283d0442fa296b420d1947e63, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952788 2023-03-24 16:55:55,610 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a67e4722ba69401f96d0da2d41efd7da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679676954047 2023-03-24 16:55:55,611 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 26573d67ab304501acb88576baa11ead, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1679676953420 2023-03-24 16:55:55,611 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b1abf2a8252e424a8f262e55ed8086d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679676954696 2023-03-24 16:55:55,611 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 0c0b4131a5624b8689b0551cdcd6e7fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679676954047 2023-03-24 16:55:55,612 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting cc21c19ad33945798b022ad768f33619, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:55,612 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 852c5d299b114d0092f76b3c53172fd6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679676954696 2023-03-24 16:55:55,612 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8f560d47a35148bfbfe971bbe5c2d695, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:55,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:55,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:55,634 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:55,649 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:55,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/cf5b16efedcf4a3181cc829185adaa25 2023-03-24 16:55:55,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677015676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,692 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:55,708 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:55,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:55,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:55,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,714 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:55,717 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/cdf461a29aab47b4ac3878203c0f32e5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cdf461a29aab47b4ac3878203c0f32e5 2023-03-24 16:55:55,722 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into cdf461a29aab47b4ac3878203c0f32e5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:55,722 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:55,722 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=11, startTime=1679676955607; duration=0sec 2023-03-24 16:55:55,722 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:55,722 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:55:55,724 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58532 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:55:55,724 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:55,724 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,724 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/34e6dd9cff7a4f54a016b6a0ee023c1e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=57.2 K 2023-03-24 16:55:55,724 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 34e6dd9cff7a4f54a016b6a0ee023c1e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679676952788 2023-03-24 16:55:55,725 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3ce614a6cfdb4c328a95d247eda1c29e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1679676953420 2023-03-24 16:55:55,725 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5d942e0184aa44ebb2a4b8dd5a2b00c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679676954047 2023-03-24 16:55:55,725 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 623b86cc28bc4b8884fd0b94257c4acb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679676954696 2023-03-24 16:55:55,726 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ffa04b1359c14da296bbfbf5f2af59dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:55,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:55,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:55,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:55,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:55,745 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:55,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,749 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8b709c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:55,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:55,751 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,752 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:55,753 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,753 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:55,753 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:55,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:55,758 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:55,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:55,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:55,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:55,758 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:55,758 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/f799846db9e74ac18c065f6d0ae9ba9c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f799846db9e74ac18c065f6d0ae9ba9c 2023-03-24 16:55:55,760 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,760 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,760 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,761 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,768 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into f799846db9e74ac18c065f6d0ae9ba9c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:55,768 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:55,768 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=11, startTime=1679676955607; duration=0sec 2023-03-24 16:55:55,769 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:55,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,772 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,773 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:55,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,773 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:55,773 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:55,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:55,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677015791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,873 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:55,879 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:55,879 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,879 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:55,879 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:55,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,882 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:55,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:55,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:55,892 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:55,892 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:55,892 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:55,892 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:55,894 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:55,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7237903e[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:55,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:55,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,899 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:55,900 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,900 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:55,902 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:55,902 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,902 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:55,902 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:55,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:55,905 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:55,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:55,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:55,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:55,905 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:55,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:55,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:55,912 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:55,918 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:55,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:55,918 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:55,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:55,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:55,919 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:55,919 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:55,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:55,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677015997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677015997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:55,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677015998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677015999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677016000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,019 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:56,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:56,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:56,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:56,028 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:56,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:56,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:56,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,030 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:56,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:56,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:56,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:56,064 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:56,064 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:56,064 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:56,065 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:56,070 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/3b48c7314721418a8a7026ce9faa7aea as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3b48c7314721418a8a7026ce9faa7aea 2023-03-24 16:55:56,074 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,074 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 3b48c7314721418a8a7026ce9faa7aea(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:56,074 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:56,075 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=11, startTime=1679676955607; duration=0sec 2023-03-24 16:55:56,075 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21dcbc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:56,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:56,077 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,078 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:56,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,078 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:56,079 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,079 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:56,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,080 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:56,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:56,082 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:56,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:56,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:56,082 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:56,082 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:56,084 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,084 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,084 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,087 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,090 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:56,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:56,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:56,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:56,094 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,095 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:56,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:56,094 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:56,095 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:56,095 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:56,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/e2fdb76549bb4eb29a1fc7f0f0016986 2023-03-24 16:55:56,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/808edd5511c24e27a47952335c2ca95a 2023-03-24 16:55:56,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/cf5b16efedcf4a3181cc829185adaa25 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25 2023-03-24 16:55:56,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25, entries=200, sequenceid=291, filesize=14.3 K 2023-03-24 16:55:56,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/e2fdb76549bb4eb29a1fc7f0f0016986 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986 2023-03-24 16:55:56,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986, entries=150, sequenceid=291, filesize=11.9 K 2023-03-24 16:55:56,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/808edd5511c24e27a47952335c2ca95a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a 2023-03-24 16:55:56,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a, entries=150, sequenceid=291, filesize=11.9 K 2023-03-24 16:55:56,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 558ms, sequenceid=291, compaction requested=false 2023-03-24 16:55:56,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:56,195 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:56,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:56,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:56,204 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:56,204 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,205 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,206 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:56,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:56,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:56,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:56,236 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:56,236 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:56,236 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:56,236 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:56,236 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,237 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:56,237 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,237 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:56,237 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:56,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:56,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677016320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677016323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677016324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677016324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677016329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677016431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677016432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677016432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677016432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677016433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677016638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677016638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677016639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677016640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677016640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,694 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/db9195f9be3e43dabbed93a3133cc21c 2023-03-24 16:55:56,711 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/893582f479ad4a3db8e51762664ece5e 2023-03-24 16:55:56,739 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/dc91334fa21c45409b61601357004ea4 2023-03-24 16:55:56,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/db9195f9be3e43dabbed93a3133cc21c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c 2023-03-24 16:55:56,751 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c, entries=150, sequenceid=312, filesize=11.9 K 2023-03-24 16:55:56,753 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/893582f479ad4a3db8e51762664ece5e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e 2023-03-24 16:55:56,766 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e, entries=150, sequenceid=312, filesize=11.9 K 2023-03-24 16:55:56,766 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/dc91334fa21c45409b61601357004ea4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4 2023-03-24 16:55:56,780 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4, entries=150, sequenceid=312, filesize=11.9 K 2023-03-24 16:55:56,781 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 545ms, sequenceid=312, compaction requested=true 2023-03-24 16:55:56,781 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:56,781 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:56,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f24a9f3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:56,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:56,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:56,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:56,789 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:56,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:56,790 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:56,792 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:56,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:56,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:56,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:56,792 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:56,793 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,793 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:56,794 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:56,795 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,796 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:56,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:56,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:56,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:56,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:56,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,802 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:56,802 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:56,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:56,802 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:56,902 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:56,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:56,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:56,906 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:56,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:56,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:56,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,907 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,909 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:56,909 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,910 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,910 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:56,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:56,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:56,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:56,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:56,921 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:56,921 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:56,921 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:56,923 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:56,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:56,956 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/89bf3cdb611b49a39c4795b76ea3a564 2023-03-24 16:55:56,978 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8140d2ea4e7c4bbe89d3e7df77ef7dca 2023-03-24 16:55:56,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677016969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677016975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677016976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677016977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:56,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:56,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677016979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677017080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677017081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677017086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677017087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677017087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677017287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677017292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677017296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677017296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677017297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,403 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/d91a50defd1a4d929dfdd7194629b7fd 2023-03-24 16:55:57,407 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/89bf3cdb611b49a39c4795b76ea3a564 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564 2023-03-24 16:55:57,411 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564, entries=150, sequenceid=326, filesize=11.9 K 2023-03-24 16:55:57,413 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8140d2ea4e7c4bbe89d3e7df77ef7dca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca 2023-03-24 16:55:57,416 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca, entries=150, sequenceid=326, filesize=11.9 K 2023-03-24 16:55:57,417 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/d91a50defd1a4d929dfdd7194629b7fd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd 2023-03-24 16:55:57,420 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd, entries=150, sequenceid=326, filesize=11.9 K 2023-03-24 16:55:57,421 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 60e0f091d1c4c40e5cc075c89fc00205 in 500ms, sequenceid=326, compaction requested=true 2023-03-24 16:55:57,421 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:57,421 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,423 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3f371b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:57,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:57,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:57,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,428 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:57,428 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:57,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:57,430 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:57,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:57,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:57,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:57,430 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:57,432 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,432 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,432 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,437 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:57,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:57,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,438 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,438 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:57,438 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:57,538 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:57,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:57,551 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:57,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:57,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:57,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:57,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:57,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:57,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:57,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:57,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,563 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:57,569 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,579 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/3198b2e818e748148c73da5940a28617 2023-03-24 16:55:57,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:57,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:57,618 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/d0a2b161eb3e4411845732f93d5a6cca 2023-03-24 16:55:57,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677017624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677017632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677017633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677017634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677017634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,658 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/f09dfa7c85d642fe8858f0086abf0724 2023-03-24 16:55:57,662 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/3198b2e818e748148c73da5940a28617 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617 2023-03-24 16:55:57,672 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617, entries=150, sequenceid=344, filesize=11.9 K 2023-03-24 16:55:57,675 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/d0a2b161eb3e4411845732f93d5a6cca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca 2023-03-24 16:55:57,681 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca, entries=150, sequenceid=344, filesize=11.9 K 2023-03-24 16:55:57,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/f09dfa7c85d642fe8858f0086abf0724 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724 2023-03-24 16:55:57,689 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724, entries=150, sequenceid=344, filesize=11.9 K 2023-03-24 16:55:57,690 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 121ms, sequenceid=344, compaction requested=true 2023-03-24 16:55:57,690 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:57,690 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:57,692 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:57,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5547e222[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:57,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:57,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,706 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:57,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:57,708 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:57,713 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:57,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:57,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:57,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:57,713 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:57,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,716 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,719 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:57,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,723 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:57,723 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:57,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:57,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/eb5b876d045f434fac85af4068178e51 2023-03-24 16:55:57,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677017768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677017777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677017782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677017783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677017785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/773b0f32f13740299b80b1d1f173d00d 2023-03-24 16:55:57,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/c0a6ace31df749a6905856fa0623c812 2023-03-24 16:55:57,824 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:57,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:57,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/eb5b876d045f434fac85af4068178e51 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51 2023-03-24 16:55:57,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51, entries=200, sequenceid=359, filesize=14.3 K 2023-03-24 16:55:57,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/773b0f32f13740299b80b1d1f173d00d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d 2023-03-24 16:55:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:57,834 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,837 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:57,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d, entries=150, sequenceid=359, filesize=11.9 K 2023-03-24 16:55:57,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/c0a6ace31df749a6905856fa0623c812 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812 2023-03-24 16:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:57,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:57,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:57,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:57,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:57,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812, entries=150, sequenceid=359, filesize=11.9 K 2023-03-24 16:55:57,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 60e0f091d1c4c40e5cc075c89fc00205 in 109ms, sequenceid=359, compaction requested=true 2023-03-24 16:55:57,852 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:57,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:57,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:57,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:57,852 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,852 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8d000a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:57,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:57,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,854 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78712 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:55:57,854 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:57,854 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,854 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cdf461a29aab47b4ac3878203c0f32e5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=76.9 K 2023-03-24 16:55:57,855 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:55:57,855 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:57,855 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,855 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3b48c7314721418a8a7026ce9faa7aea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=72.1 K 2023-03-24 16:55:57,855 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,855 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3b48c7314721418a8a7026ce9faa7aea, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:57,855 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:57,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:57,856 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting cdf461a29aab47b4ac3878203c0f32e5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:57,856 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e2fdb76549bb4eb29a1fc7f0f0016986, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679676955506 2023-03-24 16:55:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:57,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,858 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 893582f479ad4a3db8e51762664ece5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679676955669 2023-03-24 16:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,858 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting cf5b16efedcf4a3181cc829185adaa25, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679676955506 2023-03-24 16:55:57,859 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8140d2ea4e7c4bbe89d3e7df77ef7dca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679676956305 2023-03-24 16:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,859 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d0a2b161eb3e4411845732f93d5a6cca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1679676956966 2023-03-24 16:55:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,860 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting db9195f9be3e43dabbed93a3133cc21c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679676955669 2023-03-24 16:55:57,861 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 773b0f32f13740299b80b1d1f173d00d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:57,861 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 89bf3cdb611b49a39c4795b76ea3a564, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679676956305 2023-03-24 16:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,861 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3198b2e818e748148c73da5940a28617, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1679676956966 2023-03-24 16:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,862 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting eb5b876d045f434fac85af4068178e51, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:57,862 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:57,863 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:57,863 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:57,863 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:57,863 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:57,865 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,865 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,865 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:57,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:57,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,870 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:57,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,876 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:57,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:57,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:57,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:57,876 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:57,876 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:57,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:57,883 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#488 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:57,886 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#489 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:57,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:57,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:57,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677017909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,936 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/ed4b89144dc0447584b8f67ed50ef1d5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ed4b89144dc0447584b8f67ed50ef1d5 2023-03-24 16:55:57,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677017912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677017918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,940 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into ed4b89144dc0447584b8f67ed50ef1d5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:57,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677017919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,940 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:57,941 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=10, startTime=1679676957852; duration=0sec 2023-03-24 16:55:57,941 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:57,941 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:55:57,943 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:55:57,943 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:57,943 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:57,943 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f799846db9e74ac18c065f6d0ae9ba9c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=72.1 K 2023-03-24 16:55:57,944 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f799846db9e74ac18c065f6d0ae9ba9c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679676955372 2023-03-24 16:55:57,944 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 808edd5511c24e27a47952335c2ca95a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679676955506 2023-03-24 16:55:57,944 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting dc91334fa21c45409b61601357004ea4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679676955669 2023-03-24 16:55:57,945 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d91a50defd1a4d929dfdd7194629b7fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679676956305 2023-03-24 16:55:57,945 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f09dfa7c85d642fe8858f0086abf0724, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1679676956966 2023-03-24 16:55:57,945 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting c0a6ace31df749a6905856fa0623c812, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:57,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:57,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677017920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,970 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#491 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:57,977 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:57,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:57,984 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:57,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:57,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:57,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,989 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:57,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:57,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:57,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:57,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:57,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:57,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:57,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:57,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:57,999 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/82b57a5d1e49456a95e3a856f631bcdd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/82b57a5d1e49456a95e3a856f631bcdd 2023-03-24 16:55:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,004 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 82b57a5d1e49456a95e3a856f631bcdd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:58,004 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:58,004 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=10, startTime=1679676957852; duration=0sec 2023-03-24 16:55:58,004 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:58,008 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,008 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,008 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,009 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cc41b9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,015 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,015 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,017 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,017 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,017 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,019 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,019 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677018021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,034 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,041 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,041 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677018040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677018040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677018041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677018053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,141 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,144 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,146 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,146 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,147 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,156 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,156 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,156 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,156 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,158 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13a93774[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,169 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,172 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,172 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,172 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,174 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,176 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,183 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,183 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51890 deadline: 1679677018227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677018247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677018248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677018250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677018261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,283 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,286 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,288 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,290 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,290 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,290 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,290 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,293 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791591d1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,295 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,295 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,296 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,296 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,297 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,300 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,300 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,300 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,302 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,303 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,307 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,307 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,323 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/51df9dab11a74188a59d2465845d06e1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/51df9dab11a74188a59d2465845d06e1 2023-03-24 16:55:58,327 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into 51df9dab11a74188a59d2465845d06e1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:58,327 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:58,327 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=10, startTime=1679676957852; duration=0sec 2023-03-24 16:55:58,327 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:58,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/46c703376dcf4bf79881dc7e0f1887cf 2023-03-24 16:55:58,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/506f5b0e843445f592fad0a989380d78 2023-03-24 16:55:58,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/f1ac44198f9544718c5b467041651e9f 2023-03-24 16:55:58,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/46c703376dcf4bf79881dc7e0f1887cf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf 2023-03-24 16:55:58,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf, entries=200, sequenceid=377, filesize=14.3 K 2023-03-24 16:55:58,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/506f5b0e843445f592fad0a989380d78 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78 2023-03-24 16:55:58,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78, entries=150, sequenceid=377, filesize=11.9 K 2023-03-24 16:55:58,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/f1ac44198f9544718c5b467041651e9f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f 2023-03-24 16:55:58,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f, entries=150, sequenceid=377, filesize=11.9 K 2023-03-24 16:55:58,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 60e0f091d1c4c40e5cc075c89fc00205 in 489ms, sequenceid=377, compaction requested=false 2023-03-24 16:55:58,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:58,407 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,410 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,411 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,414 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,414 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,415 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:58,415 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,431 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/6b5cd6f758834ee59218a7d5c18436a5 2023-03-24 16:55:58,441 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/44bcc7937cce47a5aa211b1f166aaeef 2023-03-24 16:55:58,460 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/6f06ba756bc947ce91a708301cb6cbb7 2023-03-24 16:55:58,465 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/6b5cd6f758834ee59218a7d5c18436a5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5 2023-03-24 16:55:58,469 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5, entries=150, sequenceid=394, filesize=11.9 K 2023-03-24 16:55:58,469 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/44bcc7937cce47a5aa211b1f166aaeef as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef 2023-03-24 16:55:58,472 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef, entries=150, sequenceid=394, filesize=11.9 K 2023-03-24 16:55:58,473 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/6f06ba756bc947ce91a708301cb6cbb7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7 2023-03-24 16:55:58,484 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7, entries=150, sequenceid=394, filesize=11.9 K 2023-03-24 16:55:58,486 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 60e0f091d1c4c40e5cc075c89fc00205 in 71ms, sequenceid=394, compaction requested=true 2023-03-24 16:55:58,487 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:58,487 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,488 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a2b53a9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,505 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,505 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,507 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,507 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,511 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,511 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,512 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:58,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:55:58,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:58,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:58,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:58,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:58,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/f0ea58372b234c8cb73457f8cade6551 2023-03-24 16:55:58,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676958568, max timestamp=1679676958590Num uniques 859; 2023-03-24 16:55:58,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:58,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676958568, max timestamp=1679676958590Num uniques 859; 2023-03-24 16:55:58,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679676958568, max timestamp=1679676958590Num uniques 859; 2023-03-24 16:55:58,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:58,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:55:58,612 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,614 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,617 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,619 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,621 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677018616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597912dc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,624 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677018617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677018618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,628 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,628 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677018620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677018621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,633 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,635 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,635 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,635 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677018724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677018726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51894 deadline: 1679677018726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677018730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,736 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,739 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51876 deadline: 1679677018731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,741 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,744 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3318c10[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,748 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,748 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,748 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,750 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,752 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,752 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,754 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,758 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,758 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,858 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,862 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,865 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,866 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,866 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,866 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,866 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:58,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:58,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ff1ebe[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:58,870 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:58,870 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,871 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:58,871 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,871 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:58,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:58,877 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:58,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:58,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:58,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:58,877 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:58,879 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,879 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:58,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,879 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:58,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:58,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:58,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,884 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:58,884 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:58,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:58,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677018930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677018931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51822 deadline: 1679677018932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677018933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:58,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677018941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8aeadda766684d0686c305ddcee9f4af 2023-03-24 16:55:58,984 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:58,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:58,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:58,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/b8b29116f72a4f91b99753de10324e9c 2023-03-24 16:55:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:58,992 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:58,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:58,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:58,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,993 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:58,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:58,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:58,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:58,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/f0ea58372b234c8cb73457f8cade6551 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551 2023-03-24 16:55:58,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:58,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:58,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:58,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:58,998 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:59,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551, entries=150, sequenceid=404, filesize=11.9 K 2023-03-24 16:55:59,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/8aeadda766684d0686c305ddcee9f4af as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af 2023-03-24 16:55:59,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4659a9d3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:59,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:59,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af, entries=100, sequenceid=404, filesize=9.5 K 2023-03-24 16:55:59,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:59,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:59,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/b8b29116f72a4f91b99753de10324e9c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c 2023-03-24 16:55:59,006 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:59,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:59,009 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:59,009 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:59,009 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:59,009 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:59,009 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:59,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c, entries=100, sequenceid=404, filesize=9.5 K 2023-03-24 16:55:59,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 60e0f091d1c4c40e5cc075c89fc00205 in 454ms, sequenceid=404, compaction requested=true 2023-03-24 16:55:59,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:59,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:59,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,011 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:59,012 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:59,012 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,013 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:59,013 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:59,014 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:55:59,014 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:55:59,014 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,014 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,014 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/51df9dab11a74188a59d2465845d06e1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=50.9 K 2023-03-24 16:55:59,014 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ed4b89144dc0447584b8f67ed50ef1d5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=46.1 K 2023-03-24 16:55:59,014 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 51df9dab11a74188a59d2465845d06e1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,014 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ed4b89144dc0447584b8f67ed50ef1d5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:59,014 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 46c703376dcf4bf79881dc7e0f1887cf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679676957781 2023-03-24 16:55:59,015 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 506f5b0e843445f592fad0a989380d78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679676957782 2023-03-24 16:55:59,015 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6b5cd6f758834ee59218a7d5c18436a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1679676957905 2023-03-24 16:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,015 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 44bcc7937cce47a5aa211b1f166aaeef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1679676957905 2023-03-24 16:55:59,015 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f0ea58372b234c8cb73457f8cade6551, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676958550 2023-03-24 16:55:59,015 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8aeadda766684d0686c305ddcee9f4af, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676958553 2023-03-24 16:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,018 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:59,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:59,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:59,018 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:59,018 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:59,018 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:59,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:55:59,032 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#500 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:59,034 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#501 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:59,058 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/4f6fe23fb3b04993be9576a7da544ab5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/4f6fe23fb3b04993be9576a7da544ab5 2023-03-24 16:55:59,068 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/01ec4218880143788c4ad4aefd540758 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/01ec4218880143788c4ad4aefd540758 2023-03-24 16:55:59,070 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into 4f6fe23fb3b04993be9576a7da544ab5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:59,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:59,070 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=12, startTime=1679676959012; duration=0sec 2023-03-24 16:55:59,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:55:59,070 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:55:59,072 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:55:59,072 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:55:59,072 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,072 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/82b57a5d1e49456a95e3a856f631bcdd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=46.1 K 2023-03-24 16:55:59,073 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 82b57a5d1e49456a95e3a856f631bcdd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679676957612 2023-03-24 16:55:59,073 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f1ac44198f9544718c5b467041651e9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679676957782 2023-03-24 16:55:59,073 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 6f06ba756bc947ce91a708301cb6cbb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1679676957905 2023-03-24 16:55:59,074 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b8b29116f72a4f91b99753de10324e9c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676958553 2023-03-24 16:55:59,079 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 01ec4218880143788c4ad4aefd540758(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:59,080 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:59,080 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=12, startTime=1679676959012; duration=0sec 2023-03-24 16:55:59,080 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:59,089 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:55:59,106 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/eddda3cc3d6e4995afe8ccff6bcc1937 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eddda3cc3d6e4995afe8ccff6bcc1937 2023-03-24 16:55:59,116 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into eddda3cc3d6e4995afe8ccff6bcc1937(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:55:59,116 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:59,116 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=12, startTime=1679676959012; duration=0sec 2023-03-24 16:55:59,116 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:55:59,118 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:59,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:59,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:59,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:59,121 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:59,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:59,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:59,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:59,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:59,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:59,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:59,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:59,127 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,127 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:59,127 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 16:55:59,127 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:59,127 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:55:59,142 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/a05772517fa0414594b347f03ae8af95 2023-03-24 16:55:59,158 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/2d9dc1c4a4ad4c459586989058007007 2023-03-24 16:55:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:59,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:59,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677019247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677019247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677019249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677019250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677019251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677019353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677019354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677019357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677019357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677019558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677019558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677019560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677019561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,592 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/e6c81b8d4c6148a1bab6992796571e74 2023-03-24 16:55:59,596 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/a05772517fa0414594b347f03ae8af95 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95 2023-03-24 16:55:59,599 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95, entries=150, sequenceid=443, filesize=11.9 K 2023-03-24 16:55:59,600 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/2d9dc1c4a4ad4c459586989058007007 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007 2023-03-24 16:55:59,606 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007, entries=150, sequenceid=443, filesize=11.9 K 2023-03-24 16:55:59,606 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/e6c81b8d4c6148a1bab6992796571e74 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74 2023-03-24 16:55:59,611 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74, entries=150, sequenceid=443, filesize=11.9 K 2023-03-24 16:55:59,611 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 60e0f091d1c4c40e5cc075c89fc00205 in 484ms, sequenceid=443, compaction requested=false 2023-03-24 16:55:59,612 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:55:59,612 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:55:59,613 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,613 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57a7ac36[Count = 0] remaining members to acquire global barrier 2023-03-24 16:55:59,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:55:59,615 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:55:59,618 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:55:59,620 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:55:59,624 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:55:59,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:55:59,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:55:59,624 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:55:59,624 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:55:59,627 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,627 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:55:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:55:59,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,627 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:55:59,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:55:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:55:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:55:59,636 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:55:59,636 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,637 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:55:59,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:55:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:55:59,737 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:55:59,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:55:59,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:55:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:55:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:55:59,740 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:55:59,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:59,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:55:59,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:55:59,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,741 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,743 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:55:59,743 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:55:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,744 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,744 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:55:59,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:55:59,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:55:59,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:55:59,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:55:59,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:55:59,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:55:59,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:55:59,745 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:55:59,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:55:59,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:55:59,745 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:55:59,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:55:59,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:55:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:55:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:55:59,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:55:59,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:55:59,767 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/e039627779754f599ff69252e58016ca 2023-03-24 16:55:59,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676959812, max timestamp=1679676959833Num uniques 859; 2023-03-24 16:55:59,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:59,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676959812, max timestamp=1679676959833Num uniques 859; 2023-03-24 16:55:59,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:59,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676959812, max timestamp=1679676959833Num uniques 859; 2023-03-24 16:55:59,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:55:59,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677019874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677019875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677019882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677019883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677019884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677019985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677019985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677019986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677019991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:55:59,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:55:59,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677019991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,180 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/dcbcfa0c1867444098458d9ec7fa6d6c 2023-03-24 16:56:00,188 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/dea6b537be4b4887846e5c30d231fb81 2023-03-24 16:56:00,193 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/e039627779754f599ff69252e58016ca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca 2023-03-24 16:56:00,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677020190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677020190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677020190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,197 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca, entries=100, sequenceid=453, filesize=9.5 K 2023-03-24 16:56:00,197 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/dcbcfa0c1867444098458d9ec7fa6d6c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c 2023-03-24 16:56:00,200 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c, entries=100, sequenceid=453, filesize=9.5 K 2023-03-24 16:56:00,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,201 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/dea6b537be4b4887846e5c30d231fb81 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81 2023-03-24 16:56:00,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677020194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677020197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,206 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81, entries=100, sequenceid=453, filesize=9.5 K 2023-03-24 16:56:00,207 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 60e0f091d1c4c40e5cc075c89fc00205 in 461ms, sequenceid=453, compaction requested=true 2023-03-24 16:56:00,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:00,207 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:00,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:00,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:00,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:00,207 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:00,209 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:00,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:00,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:00,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f9ebde[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:00,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:00,211 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,212 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:00,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:00,212 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:00,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,212 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:00,214 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:00,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:00,216 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:00,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:00,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:00,216 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:00,216 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:00,218 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,218 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:00,218 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:00,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:00,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:00,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:00,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:00,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:00,226 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:00,226 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:00,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:00,226 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:00,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:00,227 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,227 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:00,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:00,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:00,326 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:00,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:00,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:00,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:00,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:00,329 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:00,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:00,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:00,329 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:00,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:00,332 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:00,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:00,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:00,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:00,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:00,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:00,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:00,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:00,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:00,334 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:00,334 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:00,344 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/898ed928d56b478493291b6c873ecb92 2023-03-24 16:56:00,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:00,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:00,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677020512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677020514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677020538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677020539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677020539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677020639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677020640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677020651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677020651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677020651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,755 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/39eb2ec3d7b842ea87a87c3a78d2e899 2023-03-24 16:56:00,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677020841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677020842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677020859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677020859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:00,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:00,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677020860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677021147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677021148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,165 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/2420fa30a0c54b9b8979be7115022f15 2023-03-24 16:56:01,169 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/898ed928d56b478493291b6c873ecb92 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92 2023-03-24 16:56:01,172 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92, entries=150, sequenceid=490, filesize=11.9 K 2023-03-24 16:56:01,173 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/39eb2ec3d7b842ea87a87c3a78d2e899 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899 2023-03-24 16:56:01,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677021167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677021167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677021167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,176 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899, entries=150, sequenceid=490, filesize=11.9 K 2023-03-24 16:56:01,177 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/2420fa30a0c54b9b8979be7115022f15 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15 2023-03-24 16:56:01,180 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15, entries=150, sequenceid=490, filesize=11.9 K 2023-03-24 16:56:01,181 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=60.38 KB/61830 for 60e0f091d1c4c40e5cc075c89fc00205 in 847ms, sequenceid=490, compaction requested=true 2023-03-24 16:56:01,181 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,181 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:01,182 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:01,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76dd1491[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:01,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:01,184 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,185 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:01,186 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,186 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:01,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:01,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:01,189 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:01,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:01,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:01,189 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:01,189 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:01,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,191 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,193 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,194 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:01,196 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:01,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:01,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:01,196 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,196 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:01,196 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:01,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:01,296 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:01,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:01,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:01,311 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:01,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:01,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:01,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:01,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,312 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,313 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:01,313 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:01,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,315 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:01,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:01,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:01,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:01,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:01,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:01,315 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:01,315 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,315 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:01,315 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:01,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,327 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/54d929deade445d487b4c4ae96f79ba3 2023-03-24 16:56:01,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:01,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:01,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677021691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677021692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677021693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677021694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677021695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,738 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/332dc5a94cf1422f8ecfc7bfcc943263 2023-03-24 16:56:01,748 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/8d6c4d126a9c48dba11733b97b9ea681 2023-03-24 16:56:01,751 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/54d929deade445d487b4c4ae96f79ba3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3 2023-03-24 16:56:01,755 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3, entries=150, sequenceid=502, filesize=11.9 K 2023-03-24 16:56:01,755 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/332dc5a94cf1422f8ecfc7bfcc943263 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263 2023-03-24 16:56:01,759 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263, entries=150, sequenceid=502, filesize=11.9 K 2023-03-24 16:56:01,759 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/8d6c4d126a9c48dba11733b97b9ea681 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681 2023-03-24 16:56:01,774 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681, entries=150, sequenceid=502, filesize=11.9 K 2023-03-24 16:56:01,775 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 60e0f091d1c4c40e5cc075c89fc00205 in 460ms, sequenceid=502, compaction requested=true 2023-03-24 16:56:01,775 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,775 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:01,777 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,777 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,778 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,778 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc12ca8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:01,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:01,779 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,781 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:01,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,782 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:01,783 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:01,785 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:01,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:01,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:01,785 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:01,785 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:01,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,787 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,789 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,792 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:01,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:01,792 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,792 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:01,792 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:01,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:01,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:01,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:01,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/3f9bf4ba019a457595116df397769f07 2023-03-24 16:56:01,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677021810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677021815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/17fb1ec17bd14a16881d023b94979c88 2023-03-24 16:56:01,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677021816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677021817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:01,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677021822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/5dba7b6dbb7b49818a5e94f6d736ce8c 2023-03-24 16:56:01,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/3f9bf4ba019a457595116df397769f07 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07 2023-03-24 16:56:01,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07, entries=150, sequenceid=523, filesize=11.9 K 2023-03-24 16:56:01,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/17fb1ec17bd14a16881d023b94979c88 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88 2023-03-24 16:56:01,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88, entries=150, sequenceid=523, filesize=11.9 K 2023-03-24 16:56:01,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/5dba7b6dbb7b49818a5e94f6d736ce8c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c 2023-03-24 16:56:01,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c, entries=150, sequenceid=523, filesize=11.9 K 2023-03-24 16:56:01,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 60e0f091d1c4c40e5cc075c89fc00205 in 50ms, sequenceid=523, compaction requested=true 2023-03-24 16:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:01,856 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:01,856 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:01,860 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71728 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:56:01,861 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71728 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:56:01,861 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:56:01,861 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:56:01,861 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,861 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,861 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/4f6fe23fb3b04993be9576a7da544ab5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=70.0 K 2023-03-24 16:56:01,861 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/01ec4218880143788c4ad4aefd540758, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=70.0 K 2023-03-24 16:56:01,861 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4f6fe23fb3b04993be9576a7da544ab5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676957905 2023-03-24 16:56:01,862 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 01ec4218880143788c4ad4aefd540758, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676957905 2023-03-24 16:56:01,862 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a05772517fa0414594b347f03ae8af95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676958604 2023-03-24 16:56:01,862 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2d9dc1c4a4ad4c459586989058007007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676958604 2023-03-24 16:56:01,862 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e039627779754f599ff69252e58016ca, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1679676959249 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting dcbcfa0c1867444098458d9ec7fa6d6c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1679676959249 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 898ed928d56b478493291b6c873ecb92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1679676959874 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 39eb2ec3d7b842ea87a87c3a78d2e899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1679676959874 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 54d929deade445d487b4c4ae96f79ba3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679676960514 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 332dc5a94cf1422f8ecfc7bfcc943263, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679676960514 2023-03-24 16:56:01,863 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3f9bf4ba019a457595116df397769f07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:01,864 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 17fb1ec17bd14a16881d023b94979c88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:01,878 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#518 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:01,880 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#519 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:01,892 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:01,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:01,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:01,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:01,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:01,902 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:01,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:01,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:01,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:01,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,902 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,905 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:01,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:01,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:01,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:01,907 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:01,907 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:01,907 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:01,907 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:01,907 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:01,907 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:01,908 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:01,908 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:01,908 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/5537c0247e444482a53c96aa1bfdecf2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5537c0247e444482a53c96aa1bfdecf2 2023-03-24 16:56:01,914 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 5537c0247e444482a53c96aa1bfdecf2(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:01,914 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,915 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=10, startTime=1679676961856; duration=0sec 2023-03-24 16:56:01,915 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:01,915 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:56:01,916 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71728 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:56:01,916 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:56:01,916 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,917 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eddda3cc3d6e4995afe8ccff6bcc1937, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=70.0 K 2023-03-24 16:56:01,917 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting eddda3cc3d6e4995afe8ccff6bcc1937, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679676957905 2023-03-24 16:56:01,919 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/77e45cff0a6c4ac09b57ddcc2aaccd81 2023-03-24 16:56:01,919 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e6c81b8d4c6148a1bab6992796571e74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679676958604 2023-03-24 16:56:01,921 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting dea6b537be4b4887846e5c30d231fb81, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1679676959249 2023-03-24 16:56:01,921 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2420fa30a0c54b9b8979be7115022f15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1679676959874 2023-03-24 16:56:01,921 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8d6c4d126a9c48dba11733b97b9ea681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679676960514 2023-03-24 16:56:01,922 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5dba7b6dbb7b49818a5e94f6d736ce8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:01,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:01,932 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/d9885a98fd7546d5a911b4af62927296 2023-03-24 16:56:01,933 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:01,947 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/2ae702b595e849b9b18de2c54d042869 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2ae702b595e849b9b18de2c54d042869 2023-03-24 16:56:01,951 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into 2ae702b595e849b9b18de2c54d042869(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:01,951 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,951 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=10, startTime=1679676961856; duration=0sec 2023-03-24 16:56:01,951 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:01,953 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/3429c09dcb6548e485c59ea0d15f9ad7 2023-03-24 16:56:01,957 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/77e45cff0a6c4ac09b57ddcc2aaccd81 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81 2023-03-24 16:56:01,961 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81, entries=150, sequenceid=534, filesize=11.9 K 2023-03-24 16:56:01,961 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/d9885a98fd7546d5a911b4af62927296 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296 2023-03-24 16:56:01,964 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296, entries=150, sequenceid=534, filesize=11.9 K 2023-03-24 16:56:01,965 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/3429c09dcb6548e485c59ea0d15f9ad7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7 2023-03-24 16:56:01,968 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7, entries=150, sequenceid=534, filesize=11.9 K 2023-03-24 16:56:01,968 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for 60e0f091d1c4c40e5cc075c89fc00205 in 60ms, sequenceid=534, compaction requested=false 2023-03-24 16:56:01,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:01,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,971 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 16:56:01,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:01,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676961960, max timestamp=1679676961970Num uniques 859; 2023-03-24 16:56:01,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:01,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676961960, max timestamp=1679676961970Num uniques 859; 2023-03-24 16:56:01,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:01,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676961960, max timestamp=1679676961970Num uniques 859; 2023-03-24 16:56:01,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:01,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:01,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:01,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ede93a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:01,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:01,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:01,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:01,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:01,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:01,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:01,981 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:01,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:01,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:01,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:01,981 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:01,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:01,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,983 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/95bafe3fa4f74f6d922d1c0180feaa8b 2023-03-24 16:56:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,990 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:01,990 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,991 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:01,991 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:01,991 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,991 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:01,991 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:01,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:01,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:01,992 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:01,992 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:01,994 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:01,994 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:01,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:01,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/74708ad1431e4c5685e3d89e2d9ac1a7 2023-03-24 16:56:02,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/555f582c6f1e46eea5ebc5541cf91959 2023-03-24 16:56:02,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/95bafe3fa4f74f6d922d1c0180feaa8b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b 2023-03-24 16:56:02,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b, entries=150, sequenceid=558, filesize=11.9 K 2023-03-24 16:56:02,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/74708ad1431e4c5685e3d89e2d9ac1a7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7 2023-03-24 16:56:02,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7, entries=150, sequenceid=558, filesize=11.9 K 2023-03-24 16:56:02,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/555f582c6f1e46eea5ebc5541cf91959 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679676962003, max timestamp=1679676962023Num uniques 859; 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679676962003, max timestamp=1679676962023Num uniques 859; 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679676962003, max timestamp=1679676962023Num uniques 859; 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8470, suffix heap size=117064,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=185 2023-03-24 16:56:02,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 16:56:02,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959, entries=150, sequenceid=558, filesize=11.9 K 2023-03-24 16:56:02,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=52.60 KB/53866 for 60e0f091d1c4c40e5cc075c89fc00205 in 72ms, sequenceid=558, compaction requested=true 2023-03-24 16:56:02,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:02,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:56:02,043 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 6 compacting, 2 eligible, 16 blocking 2023-03-24 16:56:02,044 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 16:56:02,044 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 16:56:02,044 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. because compaction request was cancelled 2023-03-24 16:56:02,044 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,045 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,045 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:56:02,045 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,045 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5537c0247e444482a53c96aa1bfdecf2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=36.9 K 2023-03-24 16:56:02,045 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5537c0247e444482a53c96aa1bfdecf2, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:02,046 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d9885a98fd7546d5a911b4af62927296, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1679676961807 2023-03-24 16:56:02,046 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 74708ad1431e4c5685e3d89e2d9ac1a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1679676961961 2023-03-24 16:56:02,050 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:02,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=59.31 KB heapSize=158.72 KB 2023-03-24 16:56:02,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:02,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,064 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/e3727677f4d74335b4e3b6b63a60b97b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e3727677f4d74335b4e3b6b63a60b97b 2023-03-24 16:56:02,068 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into e3727677f4d74335b4e3b6b63a60b97b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,068 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,068 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676962043; duration=0sec 2023-03-24 16:56:02,068 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,068 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,069 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,069 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:56:02,069 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,069 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2ae702b595e849b9b18de2c54d042869, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=36.9 K 2023-03-24 16:56:02,069 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2ae702b595e849b9b18de2c54d042869, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:02,070 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3429c09dcb6548e485c59ea0d15f9ad7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1679676961807 2023-03-24 16:56:02,070 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 555f582c6f1e46eea5ebc5541cf91959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1679676961961 2023-03-24 16:56:02,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/b348517322494b1abb9411fe7f672139 2023-03-24 16:56:02,076 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#529 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/3ec2c1926b884c508fad2d5d4c35d310 2023-03-24 16:56:02,092 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:02,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:02,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:02,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962083, max timestamp=1679676962096Num uniques 859; 2023-03-24 16:56:02,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962083, max timestamp=1679676962096Num uniques 859; 2023-03-24 16:56:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962083, max timestamp=1679676962096Num uniques 859; 2023-03-24 16:56:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:02,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.06 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/cb0bb33df71946c5b7e8fa94ad0c8860 2023-03-24 16:56:02,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:02,110 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:02,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:02,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:02,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,112 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,112 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:02,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/b348517322494b1abb9411fe7f672139 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139 2023-03-24 16:56:02,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:02,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:02,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:02,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:02,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:02,118 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/e08c2edd6f7c4df49ddbdd86d11df409 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e08c2edd6f7c4df49ddbdd86d11df409 2023-03-24 16:56:02,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139, entries=200, sequenceid=583, filesize=14.3 K 2023-03-24 16:56:02,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/3ec2c1926b884c508fad2d5d4c35d310 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310 2023-03-24 16:56:02,123 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into e08c2edd6f7c4df49ddbdd86d11df409(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,123 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,123 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676962043; duration=0sec 2023-03-24 16:56:02,123 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:02,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677022111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677022118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:02,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5702[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:02,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:02,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:02,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,128 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:02,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:02,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:02,129 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 16:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/cb0bb33df71946c5b7e8fa94ad0c8860 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860 2023-03-24 16:56:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677022119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:02,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:02,132 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:02,132 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:02,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,132 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:02,132 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:02,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677022122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,132 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:02,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 16:56:02,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~59.31 KB/60736, heapSize ~158.67 KB/162480, currentSize=134.18 KB/137400 for 60e0f091d1c4c40e5cc075c89fc00205 in 80ms, sequenceid=583, compaction requested=true 2023-03-24 16:56:02,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:56:02,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,135 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 6 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 16:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,136 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,136 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction 2023-03-24 16:56:02,136 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,136 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=38.1 K 2023-03-24 16:56:02,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,136 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 77e45cff0a6c4ac09b57ddcc2aaccd81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534 2023-03-24 16:56:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,136 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 95bafe3fa4f74f6d922d1c0180feaa8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558 2023-03-24 16:56:02,137 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b348517322494b1abb9411fe7f672139, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=583 2023-03-24 16:56:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:02,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:02,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:02,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:02,141 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:02,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,142 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:02,142 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:02,149 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#533 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677022213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677022225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677022225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677022232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,242 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:02,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:02,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677022233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:02,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:02,245 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:02,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:02,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:02,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,247 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,248 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:02,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:02,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:02,249 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,249 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:02,249 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:02,249 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:02,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:02,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3ab204[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:02,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:02,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:02,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:02,257 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:02,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:02,259 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:02,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:02,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:02,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:02,259 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:02,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,262 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,264 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,264 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,264 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:02,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,265 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,265 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,265 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:02,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,265 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,293 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/dc6268477cc44fe884aa21cc77395897 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/dc6268477cc44fe884aa21cc77395897 2023-03-24 16:56:02,298 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into dc6268477cc44fe884aa21cc77395897(size=13.1 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,298 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,298 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=10, startTime=1679676961856; duration=0sec 2023-03-24 16:56:02,298 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:56:02,299 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 16:56:02,299 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 16:56:02,299 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 16:56:02,300 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. because compaction request was cancelled 2023-03-24 16:56:02,300 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 16:56:02,300 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 16:56:02,300 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 16:56:02,301 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. because compaction request was cancelled 2023-03-24 16:56:02,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677022325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,365 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:02,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:02,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:02,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:02,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:02,369 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,370 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,370 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:02,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:02,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:02,374 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,374 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:02,374 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:02,374 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:02,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6a7907[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:02,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:02,379 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:02,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:02,380 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,380 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:02,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:02,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:02,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:02,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:02,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:02,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:02,384 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:02,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,386 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,392 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,400 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,400 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:02,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,401 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,401 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,401 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:02,401 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:02,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677022431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677022431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677022442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677022445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,501 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:02,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:02,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:02,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:02,504 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:02,504 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:02,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:02,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:02,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:02,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:02,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:02,511 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:02,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:02,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:02,513 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,513 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:02,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b6f949[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:02,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:02,516 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,518 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:02,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:02,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:02,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:02,523 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:02,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:02,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:02,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:02,523 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:02,525 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,525 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:02,525 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:02,526 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,527 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:02,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:02,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:02,529 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:02,530 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:02,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,530 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:02,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677022532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/87ce7529a360486da1b4590a0704650b 2023-03-24 16:56:02,568 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/b9656d714e0f48cead5c2d82625f320b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b9656d714e0f48cead5c2d82625f320b 2023-03-24 16:56:02,576 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into b9656d714e0f48cead5c2d82625f320b(size=12.0 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/7daffe9b31554dffb0f435b97ad951fe 2023-03-24 16:56:02,576 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,576 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=7, startTime=1679676962135; duration=0sec 2023-03-24 16:56:02,576 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:02,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/364d1088f64245b1af984615fb964b50 2023-03-24 16:56:02,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/87ce7529a360486da1b4590a0704650b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b 2023-03-24 16:56:02,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b, entries=200, sequenceid=620, filesize=14.3 K 2023-03-24 16:56:02,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/7daffe9b31554dffb0f435b97ad951fe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe 2023-03-24 16:56:02,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe, entries=150, sequenceid=620, filesize=11.9 K 2023-03-24 16:56:02,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/364d1088f64245b1af984615fb964b50 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50 2023-03-24 16:56:02,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50, entries=150, sequenceid=620, filesize=11.9 K 2023-03-24 16:56:02,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 60e0f091d1c4c40e5cc075c89fc00205 in 487ms, sequenceid=620, compaction requested=true 2023-03-24 16:56:02,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,624 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:02,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,625 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,625 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40361 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,626 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/A is initiating minor compaction (all files) 2023-03-24 16:56:02,626 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/A in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,626 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/dc6268477cc44fe884aa21cc77395897, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b9656d714e0f48cead5c2d82625f320b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=39.4 K 2023-03-24 16:56:02,626 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting dc6268477cc44fe884aa21cc77395897, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679676961692 2023-03-24 16:56:02,626 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,627 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/B is initiating minor compaction (all files) 2023-03-24 16:56:02,627 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/B in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,627 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e3727677f4d74335b4e3b6b63a60b97b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=37.0 K 2023-03-24 16:56:02,627 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b9656d714e0f48cead5c2d82625f320b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679676962005 2023-03-24 16:56:02,627 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e3727677f4d74335b4e3b6b63a60b97b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1679676961961 2023-03-24 16:56:02,628 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 87ce7529a360486da1b4590a0704650b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679676962110 2023-03-24 16:56:02,628 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3ec2c1926b884c508fad2d5d4c35d310, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679676962005 2023-03-24 16:56:02,628 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7daffe9b31554dffb0f435b97ad951fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679676962110 2023-03-24 16:56:02,630 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:02,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:02,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:02,633 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:02,633 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:02,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:02,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,634 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,637 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:02,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:02,637 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#A#compaction#536 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,637 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#B#compaction#537 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:02,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:02,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:02,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:02,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:02,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:02,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:02,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:02,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:02,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:02,647 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:02,648 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,648 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:02,648 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:02,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:02,659 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/5a99511a147744908ef95b7466fa4354 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5a99511a147744908ef95b7466fa4354 2023-03-24 16:56:02,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/B of 60e0f091d1c4c40e5cc075c89fc00205 into 5a99511a147744908ef95b7466fa4354(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/B, priority=13, startTime=1679676962624; duration=0sec 2023-03-24 16:56:02,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:02,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:02,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:02,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): 60e0f091d1c4c40e5cc075c89fc00205/C is initiating minor compaction (all files) 2023-03-24 16:56:02,665 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 60e0f091d1c4c40e5cc075c89fc00205/C in TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:02,665 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e08c2edd6f7c4df49ddbdd86d11df409, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp, totalSize=37.0 K 2023-03-24 16:56:02,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e08c2edd6f7c4df49ddbdd86d11df409, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1679676961961 2023-03-24 16:56:02,666 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cb0bb33df71946c5b7e8fa94ad0c8860, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679676962005 2023-03-24 16:56:02,666 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 364d1088f64245b1af984615fb964b50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679676962110 2023-03-24 16:56:02,673 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): 60e0f091d1c4c40e5cc075c89fc00205#C#compaction#539 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:02,686 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/d174d29747a44b82b807a952e093f80c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d174d29747a44b82b807a952e093f80c 2023-03-24 16:56:02,697 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/C of 60e0f091d1c4c40e5cc075c89fc00205 into d174d29747a44b82b807a952e093f80c(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:02,697 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:02,697 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/C, priority=13, startTime=1679676962625; duration=0sec 2023-03-24 16:56:02,697 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:02,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:02,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. as already flushing 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962752, max timestamp=1679676962767Num uniques 859; 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962752, max timestamp=1679676962767Num uniques 859; 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679676962752, max timestamp=1679676962767Num uniques 859; 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 16:56:02,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677022797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677022798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677022798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677022799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677022833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677022900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677022905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677022906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:02,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677022906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,056 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/f1bf2c7a10804fe4b1dab4f64234f057 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f1bf2c7a10804fe4b1dab4f64234f057 2023-03-24 16:56:03,060 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 60e0f091d1c4c40e5cc075c89fc00205/A of 60e0f091d1c4c40e5cc075c89fc00205 into f1bf2c7a10804fe4b1dab4f64234f057(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:03,060 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:03,060 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205., storeName=60e0f091d1c4c40e5cc075c89fc00205/A, priority=13, startTime=1679676962624; duration=0sec 2023-03-24 16:56:03,060 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:03,069 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/5aba7c51f28145188548d58715d019f7 2023-03-24 16:56:03,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677023106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677023110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677023111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677023111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51876 deadline: 1679677023343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51850 deadline: 1679677023409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51894 deadline: 1679677023419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51890 deadline: 1679677023419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:03,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51822 deadline: 1679677023421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,482 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/99154db9651e498681eb08398383efae 2023-03-24 16:56:03,492 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/886e9e2d75ba43ca9c2eaab085831157 2023-03-24 16:56:03,496 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/5aba7c51f28145188548d58715d019f7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5aba7c51f28145188548d58715d019f7 2023-03-24 16:56:03,500 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5aba7c51f28145188548d58715d019f7, entries=150, sequenceid=632, filesize=11.9 K 2023-03-24 16:56:03,501 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/99154db9651e498681eb08398383efae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/99154db9651e498681eb08398383efae 2023-03-24 16:56:03,503 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/99154db9651e498681eb08398383efae, entries=150, sequenceid=632, filesize=11.9 K 2023-03-24 16:56:03,504 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/886e9e2d75ba43ca9c2eaab085831157 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/886e9e2d75ba43ca9c2eaab085831157 2023-03-24 16:56:03,507 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/886e9e2d75ba43ca9c2eaab085831157, entries=150, sequenceid=632, filesize=11.9 K 2023-03-24 16:56:03,508 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 60e0f091d1c4c40e5cc075c89fc00205 in 860ms, sequenceid=632, compaction requested=false 2023-03-24 16:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b9495a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:03,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:03,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:03,513 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,513 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:03,514 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:03,518 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:03,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:03,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:03,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:03,518 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:03,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:03,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:03,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:03,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:03,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,525 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:03,525 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,625 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:03,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:03,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:03,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:03,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:03,627 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:03,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:03,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:03,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:03,631 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:03,632 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:03,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:03,648 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/24206f22137941cf9e85f8f95b147a7a 2023-03-24 16:56:03,658 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/f731c75400d5488089f9b6b644fcc5fc 2023-03-24 16:56:03,666 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/71809ae305c54bdc999d6c4df44a6a40 2023-03-24 16:56:03,670 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/24206f22137941cf9e85f8f95b147a7a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/24206f22137941cf9e85f8f95b147a7a 2023-03-24 16:56:03,673 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/24206f22137941cf9e85f8f95b147a7a, entries=150, sequenceid=670, filesize=11.9 K 2023-03-24 16:56:03,673 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/f731c75400d5488089f9b6b644fcc5fc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f731c75400d5488089f9b6b644fcc5fc 2023-03-24 16:56:03,676 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f731c75400d5488089f9b6b644fcc5fc, entries=150, sequenceid=670, filesize=11.9 K 2023-03-24 16:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/71809ae305c54bdc999d6c4df44a6a40 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/71809ae305c54bdc999d6c4df44a6a40 2023-03-24 16:56:03,681 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/71809ae305c54bdc999d6c4df44a6a40, entries=150, sequenceid=670, filesize=11.9 K 2023-03-24 16:56:03,681 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 60e0f091d1c4c40e5cc075c89fc00205 in 49ms, sequenceid=670, compaction requested=true 2023-03-24 16:56:03,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:03,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:03,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:03,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:03,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:03,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d87edb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:03,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:03,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:03,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:03,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:03,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:03,689 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,689 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:03,689 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,689 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:03,691 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:03,691 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:03,691 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:03,692 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:03,692 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:03,693 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,693 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,693 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:03,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:03,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:03,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:03,699 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:03,699 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:03,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,799 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:03,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:03,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:03,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:03,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:03,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:03,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:03,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:03,804 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,804 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,808 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,808 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:03,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:03,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,809 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:03,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:03,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:03,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:03,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. started... 2023-03-24 16:56:03,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:03,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:03,815 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac38be6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:03,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:03,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:03,821 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:03,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:03,821 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:03,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:03,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,823 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,824 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:03,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:03,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:03,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:03,829 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:03,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:03,829 DEBUG [Thread-3257] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:03,830 DEBUG [Thread-3257] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:03,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:03,898 DEBUG [Thread-3258] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:62195 2023-03-24 16:56:03,898 DEBUG [Thread-3264] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:62195 2023-03-24 16:56:03,898 DEBUG [Thread-3258] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,898 DEBUG [Thread-3264] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,900 DEBUG [Thread-3262] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:62195 2023-03-24 16:56:03,900 DEBUG [Thread-3262] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,901 DEBUG [Thread-3260] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:62195 2023-03-24 16:56:03,902 DEBUG [Thread-3260] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,902 DEBUG [Thread-3266] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:62195 2023-03-24 16:56:03,902 DEBUG [Thread-3266] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,921 DEBUG [Thread-3255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:62195 2023-03-24 16:56:03,921 DEBUG [Thread-3255] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,928 DEBUG [Thread-3251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:62195 2023-03-24 16:56:03,928 DEBUG [Thread-3251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,930 DEBUG [Thread-3257] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:03,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:03,931 DEBUG [Thread-3247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a987907 to 127.0.0.1:62195 2023-03-24 16:56:03,931 DEBUG [Thread-3247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:03,932 DEBUG [Thread-3253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:62195 2023-03-24 16:56:03,932 DEBUG [Thread-3253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:04,360 DEBUG [Thread-3249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:62195 2023-03-24 16:56:04,360 DEBUG [Thread-3249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:04,360 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:56:04,360 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 189 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1963 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5882 rows 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1980 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5936 rows 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1964 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5888 rows 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1985 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5953 rows 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(402): scanned 1958 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(403): verified 5866 rows 2023-03-24 16:56:04,361 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:56:04,361 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73eb7080 to 127.0.0.1:62195 2023-03-24 16:56:04,361 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:04,365 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:56:04,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:56:04,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:04,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:04,369 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676964369"}]},"ts":"1679676964369"} 2023-03-24 16:56:04,370 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:56:04,372 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:56:04,373 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, UNASSIGN}] 2023-03-24 16:56:04,374 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, UNASSIGN 2023-03-24 16:56:04,375 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=60e0f091d1c4c40e5cc075c89fc00205, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:04,375 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676964375"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676964375"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676964375"}]},"ts":"1679676964375"} 2023-03-24 16:56:04,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:56:04,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:04,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 60e0f091d1c4c40e5cc075c89fc00205, disabling compactions & flushes 2023-03-24 16:56:04,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. after waiting 0 ms 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:04,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 60e0f091d1c4c40e5cc075c89fc00205 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=A 2023-03-24 16:56:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:04,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=B 2023-03-24 16:56:04,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:04,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 60e0f091d1c4c40e5cc075c89fc00205, store=C 2023-03-24 16:56:04,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:04,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:04,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/86cedf1ffaf0467f830c5213b64a1eef 2023-03-24 16:56:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:05,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/515682f38bf3464882043f01849c959a 2023-03-24 16:56:05,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:05,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/ba1e55a79f974d0cb04d54669cdc1071 2023-03-24 16:56:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/A/86cedf1ffaf0467f830c5213b64a1eef as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/86cedf1ffaf0467f830c5213b64a1eef 2023-03-24 16:56:05,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/86cedf1ffaf0467f830c5213b64a1eef, entries=100, sequenceid=679, filesize=9.5 K 2023-03-24 16:56:05,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/B/515682f38bf3464882043f01849c959a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/515682f38bf3464882043f01849c959a 2023-03-24 16:56:05,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/515682f38bf3464882043f01849c959a, entries=100, sequenceid=679, filesize=9.5 K 2023-03-24 16:56:05,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/.tmp/C/ba1e55a79f974d0cb04d54669cdc1071 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ba1e55a79f974d0cb04d54669cdc1071 2023-03-24 16:56:05,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ba1e55a79f974d0cb04d54669cdc1071, entries=100, sequenceid=679, filesize=9.5 K 2023-03-24 16:56:05,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 60e0f091d1c4c40e5cc075c89fc00205 in 1237ms, sequenceid=679, compaction requested=true 2023-03-24 16:56:05,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/ddf65a12e55343d9b45ee54da8082829, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/02e14a197b0a48939935e1063de4a16f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0816675fb4df4e5ca0d20be035c93631, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a31650bdd940446aa25e7f97b2d15624, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bd485102054a433d99a204d4ddc7e3dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cdf461a29aab47b4ac3878203c0f32e5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/51df9dab11a74188a59d2465845d06e1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/4f6fe23fb3b04993be9576a7da544ab5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/dc6268477cc44fe884aa21cc77395897, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b9656d714e0f48cead5c2d82625f320b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b] to archive 2023-03-24 16:56:05,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:05,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5feb780502bc4743acf57a3c004aef90 2023-03-24 16:56:05,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/aa3ecaf3ce784e0c9dd960c5615fff5b 2023-03-24 16:56:05,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/55678991d03f40e78cdedab908038dde 2023-03-24 16:56:05,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/ddf65a12e55343d9b45ee54da8082829 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/ddf65a12e55343d9b45ee54da8082829 2023-03-24 16:56:05,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d4a8b80475e041fb81dee936111d6bd0 2023-03-24 16:56:05,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/d9a8888020e548468bc749d467354ab7 2023-03-24 16:56:05,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/02e14a197b0a48939935e1063de4a16f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/02e14a197b0a48939935e1063de4a16f 2023-03-24 16:56:05,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f352ab83befe4a0c9878ce2282ea59e0 2023-03-24 16:56:05,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/c808b5238b6743d0acdb8f236650e627 2023-03-24 16:56:05,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0816675fb4df4e5ca0d20be035c93631 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0816675fb4df4e5ca0d20be035c93631 2023-03-24 16:56:05,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95459c36f1da49c0ae7f8f47039a6b80 2023-03-24 16:56:05,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/22101baaa0824175a8f4edde8cb7884c 2023-03-24 16:56:05,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/0a3cd7035922453685f373fa4f2c068b 2023-03-24 16:56:05,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a31650bdd940446aa25e7f97b2d15624 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a31650bdd940446aa25e7f97b2d15624 2023-03-24 16:56:05,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/14c02e971749404bb0c84bc14e96dfbc 2023-03-24 16:56:05,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6a5b91802adc41ff90a8369fec467b2e 2023-03-24 16:56:05,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bd485102054a433d99a204d4ddc7e3dc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bd485102054a433d99a204d4ddc7e3dc 2023-03-24 16:56:05,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/bb88a0570a6a414a8f3969d0df2f3fdc 2023-03-24 16:56:05,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a67e4722ba69401f96d0da2d41efd7da 2023-03-24 16:56:05,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b1abf2a8252e424a8f262e55ed8086d3 2023-03-24 16:56:05,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cc21c19ad33945798b022ad768f33619 2023-03-24 16:56:05,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cdf461a29aab47b4ac3878203c0f32e5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cdf461a29aab47b4ac3878203c0f32e5 2023-03-24 16:56:05,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/cf5b16efedcf4a3181cc829185adaa25 2023-03-24 16:56:05,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/db9195f9be3e43dabbed93a3133cc21c 2023-03-24 16:56:05,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/89bf3cdb611b49a39c4795b76ea3a564 2023-03-24 16:56:05,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3198b2e818e748148c73da5940a28617 2023-03-24 16:56:05,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/eb5b876d045f434fac85af4068178e51 2023-03-24 16:56:05,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/51df9dab11a74188a59d2465845d06e1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/51df9dab11a74188a59d2465845d06e1 2023-03-24 16:56:05,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/46c703376dcf4bf79881dc7e0f1887cf 2023-03-24 16:56:05,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/6b5cd6f758834ee59218a7d5c18436a5 2023-03-24 16:56:05,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/4f6fe23fb3b04993be9576a7da544ab5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/4f6fe23fb3b04993be9576a7da544ab5 2023-03-24 16:56:05,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f0ea58372b234c8cb73457f8cade6551 2023-03-24 16:56:05,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/a05772517fa0414594b347f03ae8af95 2023-03-24 16:56:05,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/e039627779754f599ff69252e58016ca 2023-03-24 16:56:05,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/898ed928d56b478493291b6c873ecb92 2023-03-24 16:56:05,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/54d929deade445d487b4c4ae96f79ba3 2023-03-24 16:56:05,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/dc6268477cc44fe884aa21cc77395897 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/dc6268477cc44fe884aa21cc77395897 2023-03-24 16:56:05,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/3f9bf4ba019a457595116df397769f07 2023-03-24 16:56:05,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/77e45cff0a6c4ac09b57ddcc2aaccd81 2023-03-24 16:56:05,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/95bafe3fa4f74f6d922d1c0180feaa8b 2023-03-24 16:56:05,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b348517322494b1abb9411fe7f672139 2023-03-24 16:56:05,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b9656d714e0f48cead5c2d82625f320b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/b9656d714e0f48cead5c2d82625f320b 2023-03-24 16:56:05,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/87ce7529a360486da1b4590a0704650b 2023-03-24 16:56:05,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/4a0cfa68b97f444fb716907454d286c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f00a68ef715441e8aaa4a95f0ee50e0a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/1fdad176a491427c926afc053a1d4e1c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8eba811ebe6c422290519822edb3a780, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3dcf8a6283d0442fa296b420d1947e63, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3b48c7314721418a8a7026ce9faa7aea, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ed4b89144dc0447584b8f67ed50ef1d5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/01ec4218880143788c4ad4aefd540758, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5537c0247e444482a53c96aa1bfdecf2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e3727677f4d74335b4e3b6b63a60b97b, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe] to archive 2023-03-24 16:56:05,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:05,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/83b1378ab79549dabc803313efcf3b70 2023-03-24 16:56:05,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a9cefc7247c144978f55d6f239c00fcd 2023-03-24 16:56:05,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/4a0cfa68b97f444fb716907454d286c3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/4a0cfa68b97f444fb716907454d286c3 2023-03-24 16:56:05,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/78212be0d0a6455c86b937f00af127a9 2023-03-24 16:56:05,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/caddc839ad7b46a7b4f4462f3393c7db 2023-03-24 16:56:05,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f00a68ef715441e8aaa4a95f0ee50e0a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f00a68ef715441e8aaa4a95f0ee50e0a 2023-03-24 16:56:05,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ef998687799a4ec58deb3bc5a9aa3f17 2023-03-24 16:56:05,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/a52d5007ddc342dfb6d046b1bfda470c 2023-03-24 16:56:05,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/1fdad176a491427c926afc053a1d4e1c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/1fdad176a491427c926afc053a1d4e1c 2023-03-24 16:56:05,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8717b88f6e4a4158ada6e58422da2507 2023-03-24 16:56:05,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/70245985054a46f9918248a6e40f76c3 2023-03-24 16:56:05,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/b3fa0927c1104ecd8fd23303cf991a1d 2023-03-24 16:56:05,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8eba811ebe6c422290519822edb3a780 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8eba811ebe6c422290519822edb3a780 2023-03-24 16:56:05,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0beee4141b7f41449ec8521bbb19270c 2023-03-24 16:56:05,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/c211aa53dfb34a9bbb6b5f84cf424867 2023-03-24 16:56:05,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3dcf8a6283d0442fa296b420d1947e63 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3dcf8a6283d0442fa296b420d1947e63 2023-03-24 16:56:05,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/359cc6f8a3b344d48ce58df3a1501685 2023-03-24 16:56:05,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/26573d67ab304501acb88576baa11ead 2023-03-24 16:56:05,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/0c0b4131a5624b8689b0551cdcd6e7fc 2023-03-24 16:56:05,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/852c5d299b114d0092f76b3c53172fd6 2023-03-24 16:56:05,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3b48c7314721418a8a7026ce9faa7aea to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3b48c7314721418a8a7026ce9faa7aea 2023-03-24 16:56:05,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8f560d47a35148bfbfe971bbe5c2d695 2023-03-24 16:56:05,861 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e2fdb76549bb4eb29a1fc7f0f0016986 2023-03-24 16:56:05,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/893582f479ad4a3db8e51762664ece5e 2023-03-24 16:56:05,863 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8140d2ea4e7c4bbe89d3e7df77ef7dca 2023-03-24 16:56:05,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d0a2b161eb3e4411845732f93d5a6cca 2023-03-24 16:56:05,865 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ed4b89144dc0447584b8f67ed50ef1d5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/ed4b89144dc0447584b8f67ed50ef1d5 2023-03-24 16:56:05,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/773b0f32f13740299b80b1d1f173d00d 2023-03-24 16:56:05,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/506f5b0e843445f592fad0a989380d78 2023-03-24 16:56:05,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/44bcc7937cce47a5aa211b1f166aaeef 2023-03-24 16:56:05,869 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/01ec4218880143788c4ad4aefd540758 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/01ec4218880143788c4ad4aefd540758 2023-03-24 16:56:05,870 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/8aeadda766684d0686c305ddcee9f4af 2023-03-24 16:56:05,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/2d9dc1c4a4ad4c459586989058007007 2023-03-24 16:56:05,872 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/dcbcfa0c1867444098458d9ec7fa6d6c 2023-03-24 16:56:05,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/39eb2ec3d7b842ea87a87c3a78d2e899 2023-03-24 16:56:05,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/332dc5a94cf1422f8ecfc7bfcc943263 2023-03-24 16:56:05,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5537c0247e444482a53c96aa1bfdecf2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5537c0247e444482a53c96aa1bfdecf2 2023-03-24 16:56:05,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/17fb1ec17bd14a16881d023b94979c88 2023-03-24 16:56:05,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/d9885a98fd7546d5a911b4af62927296 2023-03-24 16:56:05,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e3727677f4d74335b4e3b6b63a60b97b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/e3727677f4d74335b4e3b6b63a60b97b 2023-03-24 16:56:05,878 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/74708ad1431e4c5685e3d89e2d9ac1a7 2023-03-24 16:56:05,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/3ec2c1926b884c508fad2d5d4c35d310 2023-03-24 16:56:05,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/7daffe9b31554dffb0f435b97ad951fe 2023-03-24 16:56:05,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/4cd0001f5e044fe0ba48f0d7084581cb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/11552c00b1af45bc945ba65c11e32249, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3290e76a514b4d88b4abaa22d1130271, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d37a45deb8a04e3aa0008bbb4977ae30, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/34e6dd9cff7a4f54a016b6a0ee023c1e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f799846db9e74ac18c065f6d0ae9ba9c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/82b57a5d1e49456a95e3a856f631bcdd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eddda3cc3d6e4995afe8ccff6bcc1937, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2ae702b595e849b9b18de2c54d042869, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e08c2edd6f7c4df49ddbdd86d11df409, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50] to archive 2023-03-24 16:56:05,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:05,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e43ed1c267dc4738be91d4ee73acbe7a 2023-03-24 16:56:05,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/37709b7d6c204385aa7f3b23e784759e 2023-03-24 16:56:05,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/4cd0001f5e044fe0ba48f0d7084581cb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/4cd0001f5e044fe0ba48f0d7084581cb 2023-03-24 16:56:05,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/25fb0cd64405469d9fb6969f28aec9ce 2023-03-24 16:56:05,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8efdd1e0fde04058b13108d47ca734a6 2023-03-24 16:56:05,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/11552c00b1af45bc945ba65c11e32249 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/11552c00b1af45bc945ba65c11e32249 2023-03-24 16:56:05,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/380cbe7367564960a3b4feb14df08869 2023-03-24 16:56:05,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/013a07e884d14065b33155a999578eb0 2023-03-24 16:56:05,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3290e76a514b4d88b4abaa22d1130271 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3290e76a514b4d88b4abaa22d1130271 2023-03-24 16:56:05,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b4a5bc2f7f734f87b262109cbfe1f4d9 2023-03-24 16:56:05,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/9e42f73d872549edb1a756b89b55994d 2023-03-24 16:56:05,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/63085aa6bdbb436ba1967bda6a299503 2023-03-24 16:56:05,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d37a45deb8a04e3aa0008bbb4977ae30 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d37a45deb8a04e3aa0008bbb4977ae30 2023-03-24 16:56:05,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/591d49d64b194bec8b7bd54859c7cfbb 2023-03-24 16:56:05,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/de53b808f4fa42ceb8bc5876db496975 2023-03-24 16:56:05,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/34e6dd9cff7a4f54a016b6a0ee023c1e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/34e6dd9cff7a4f54a016b6a0ee023c1e 2023-03-24 16:56:05,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eeabc83c9f1a4ce1bc92b5063d1b819d 2023-03-24 16:56:05,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3ce614a6cfdb4c328a95d247eda1c29e 2023-03-24 16:56:05,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5d942e0184aa44ebb2a4b8dd5a2b00c9 2023-03-24 16:56:05,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/623b86cc28bc4b8884fd0b94257c4acb 2023-03-24 16:56:05,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f799846db9e74ac18c065f6d0ae9ba9c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f799846db9e74ac18c065f6d0ae9ba9c 2023-03-24 16:56:05,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ffa04b1359c14da296bbfbf5f2af59dc 2023-03-24 16:56:05,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/808edd5511c24e27a47952335c2ca95a 2023-03-24 16:56:05,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dc91334fa21c45409b61601357004ea4 2023-03-24 16:56:05,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d91a50defd1a4d929dfdd7194629b7fd 2023-03-24 16:56:05,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f09dfa7c85d642fe8858f0086abf0724 2023-03-24 16:56:05,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/82b57a5d1e49456a95e3a856f631bcdd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/82b57a5d1e49456a95e3a856f631bcdd 2023-03-24 16:56:05,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/c0a6ace31df749a6905856fa0623c812 2023-03-24 16:56:05,911 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/f1ac44198f9544718c5b467041651e9f 2023-03-24 16:56:05,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/6f06ba756bc947ce91a708301cb6cbb7 2023-03-24 16:56:05,913 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eddda3cc3d6e4995afe8ccff6bcc1937 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/eddda3cc3d6e4995afe8ccff6bcc1937 2023-03-24 16:56:05,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/b8b29116f72a4f91b99753de10324e9c 2023-03-24 16:56:05,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e6c81b8d4c6148a1bab6992796571e74 2023-03-24 16:56:05,916 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/dea6b537be4b4887846e5c30d231fb81 2023-03-24 16:56:05,917 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2420fa30a0c54b9b8979be7115022f15 2023-03-24 16:56:05,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/8d6c4d126a9c48dba11733b97b9ea681 2023-03-24 16:56:05,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2ae702b595e849b9b18de2c54d042869 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/2ae702b595e849b9b18de2c54d042869 2023-03-24 16:56:05,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/5dba7b6dbb7b49818a5e94f6d736ce8c 2023-03-24 16:56:05,920 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/3429c09dcb6548e485c59ea0d15f9ad7 2023-03-24 16:56:05,921 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e08c2edd6f7c4df49ddbdd86d11df409 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/e08c2edd6f7c4df49ddbdd86d11df409 2023-03-24 16:56:05,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/555f582c6f1e46eea5ebc5541cf91959 2023-03-24 16:56:05,923 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/cb0bb33df71946c5b7e8fa94ad0c8860 2023-03-24 16:56:05,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/364d1088f64245b1af984615fb964b50 2023-03-24 16:56:05,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/recovered.edits/682.seqid, newMaxSeqId=682, maxSeqId=1 2023-03-24 16:56:05,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205. 2023-03-24 16:56:05,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 60e0f091d1c4c40e5cc075c89fc00205: 2023-03-24 16:56:05,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:05,931 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=60e0f091d1c4c40e5cc075c89fc00205, regionState=CLOSED 2023-03-24 16:56:05,931 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676965931"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676965931"}]},"ts":"1679676965931"} 2023-03-24 16:56:05,933 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-24 16:56:05,933 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 60e0f091d1c4c40e5cc075c89fc00205, server=jenkins-hbase4.apache.org,33315,1679676819242 in 1.5570 sec 2023-03-24 16:56:05,934 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-24 16:56:05,934 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=60e0f091d1c4c40e5cc075c89fc00205, UNASSIGN in 1.5600 sec 2023-03-24 16:56:05,935 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676965935"}]},"ts":"1679676965935"} 2023-03-24 16:56:05,935 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:56:05,937 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:56:05,938 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5730 sec 2023-03-24 16:56:06,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 16:56:06,474 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-24 16:56:06,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:56:06,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,476 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 16:56:06,477 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,481 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:06,483 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/recovered.edits] 2023-03-24 16:56:06,485 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/24206f22137941cf9e85f8f95b147a7a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/24206f22137941cf9e85f8f95b147a7a 2023-03-24 16:56:06,486 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5aba7c51f28145188548d58715d019f7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/5aba7c51f28145188548d58715d019f7 2023-03-24 16:56:06,487 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/86cedf1ffaf0467f830c5213b64a1eef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/86cedf1ffaf0467f830c5213b64a1eef 2023-03-24 16:56:06,488 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f1bf2c7a10804fe4b1dab4f64234f057 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/A/f1bf2c7a10804fe4b1dab4f64234f057 2023-03-24 16:56:06,490 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/515682f38bf3464882043f01849c959a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/515682f38bf3464882043f01849c959a 2023-03-24 16:56:06,491 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5a99511a147744908ef95b7466fa4354 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/5a99511a147744908ef95b7466fa4354 2023-03-24 16:56:06,492 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/99154db9651e498681eb08398383efae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/99154db9651e498681eb08398383efae 2023-03-24 16:56:06,493 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f731c75400d5488089f9b6b644fcc5fc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/B/f731c75400d5488089f9b6b644fcc5fc 2023-03-24 16:56:06,494 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/71809ae305c54bdc999d6c4df44a6a40 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/71809ae305c54bdc999d6c4df44a6a40 2023-03-24 16:56:06,495 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/886e9e2d75ba43ca9c2eaab085831157 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/886e9e2d75ba43ca9c2eaab085831157 2023-03-24 16:56:06,496 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ba1e55a79f974d0cb04d54669cdc1071 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/ba1e55a79f974d0cb04d54669cdc1071 2023-03-24 16:56:06,497 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d174d29747a44b82b807a952e093f80c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/C/d174d29747a44b82b807a952e093f80c 2023-03-24 16:56:06,499 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/recovered.edits/682.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205/recovered.edits/682.seqid 2023-03-24 16:56:06,499 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/60e0f091d1c4c40e5cc075c89fc00205 2023-03-24 16:56:06,499 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:56:06,501 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,502 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:56:06,503 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:56:06,504 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,504 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:56:06,504 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676966504"}]},"ts":"9223372036854775807"} 2023-03-24 16:56:06,506 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:56:06,506 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 60e0f091d1c4c40e5cc075c89fc00205, NAME => 'TestAcidGuarantees,,1679676941729.60e0f091d1c4c40e5cc075c89fc00205.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:56:06,506 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:56:06,506 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676966506"}]},"ts":"9223372036854775807"} 2023-03-24 16:56:06,507 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:56:06,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-03-24 16:56:06,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 16:56:06,578 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-24 16:56:06,589 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=233 (was 226) - Thread LEAK? -, OpenFileDescriptor=606 (was 508) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=361 (was 315) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 169), AvailableMemoryMB=5690 (was 5447) - AvailableMemoryMB LEAK? - 2023-03-24 16:56:06,604 INFO [Listener at localhost/43391] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=233, OpenFileDescriptor=606, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=168, AvailableMemoryMB=5689 2023-03-24 16:56:06,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:56:06,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:56:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:06,608 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 16:56:06,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-24 16:56:06,609 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 16:56:06,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:06,610 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:06,610 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 empty. 2023-03-24 16:56:06,611 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:06,611 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:56:06,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:06,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:07,017 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:56:07,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fa8f591d6f09d5e6894f62e442177704, NAME => 'TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp 2023-03-24 16:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fa8f591d6f09d5e6894f62e442177704, disabling compactions & flushes 2023-03-24 16:56:07,430 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. after waiting 0 ms 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,430 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:07,432 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 16:56:07,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676967433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676967433"}]},"ts":"1679676967433"} 2023-03-24 16:56:07,434 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 16:56:07,434 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 16:56:07,435 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676967435"}]},"ts":"1679676967435"} 2023-03-24 16:56:07,435 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 16:56:07,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, ASSIGN}] 2023-03-24 16:56:07,440 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, ASSIGN 2023-03-24 16:56:07,441 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=false 2023-03-24 16:56:07,592 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:07,593 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676967592"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676967592"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676967592"}]},"ts":"1679676967592"} 2023-03-24 16:56:07,594 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:56:07,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:07,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fa8f591d6f09d5e6894f62e442177704, NAME => 'TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:56:07,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:56:07,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,749 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,750 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A 2023-03-24 16:56:07,750 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A 2023-03-24 16:56:07,750 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:07,751 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName A 2023-03-24 16:56:07,751 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:07,751 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,752 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B 2023-03-24 16:56:07,752 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B 2023-03-24 16:56:07,752 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:07,752 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName B 2023-03-24 16:56:07,753 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:07,753 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,754 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C 2023-03-24 16:56:07,754 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C 2023-03-24 16:56:07,754 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:07,754 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName C 2023-03-24 16:56:07,755 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:07,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:56:07,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:07,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 16:56:07,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fa8f591d6f09d5e6894f62e442177704; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71912387, jitterRate=0.07157807052135468}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:56:07,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:07,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., pid=56, masterSystemTime=1679676967745 2023-03-24 16:56:07,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:07,762 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:07,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676967762"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676967762"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676967762"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676967762"}]},"ts":"1679676967762"} 2023-03-24 16:56:07,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-24 16:56:07,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 in 169 msec 2023-03-24 16:56:07,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-24 16:56:07,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, ASSIGN in 326 msec 2023-03-24 16:56:07,765 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 16:56:07,766 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676967765"}]},"ts":"1679676967765"} 2023-03-24 16:56:07,766 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 16:56:07,768 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 16:56:07,769 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-03-24 16:56:08,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 16:56:08,713 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-24 16:56:08,714 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:08,720 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:08,721 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:08,722 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46830, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:08,723 DEBUG [Listener at localhost/43391] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 16:56:08,725 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51880, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 16:56:08,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 16:56:08,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 16:56:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:09,138 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:56:09,139 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 16:56:09,139 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 16:56:09,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 16:56:09,143 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, REOPEN/MOVE}] 2023-03-24 16:56:09,144 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, REOPEN/MOVE 2023-03-24 16:56:09,144 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,144 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676969144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676969144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676969144"}]},"ts":"1679676969144"} 2023-03-24 16:56:09,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:56:09,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fa8f591d6f09d5e6894f62e442177704, disabling compactions & flushes 2023-03-24 16:56:09,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. after waiting 0 ms 2023-03-24 16:56:09,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 16:56:09,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:09,302 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fa8f591d6f09d5e6894f62e442177704 to self. 2023-03-24 16:56:09,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,303 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=CLOSED 2023-03-24 16:56:09,303 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676969303"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676969303"}]},"ts":"1679676969303"} 2023-03-24 16:56:09,305 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-24 16:56:09,305 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 in 159 msec 2023-03-24 16:56:09,305 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33315,1679676819242; forceNewPlan=false, retain=true 2023-03-24 16:56:09,456 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,456 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676969456"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676969456"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676969456"}]},"ts":"1679676969456"} 2023-03-24 16:56:09,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:56:09,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fa8f591d6f09d5e6894f62e442177704, NAME => 'TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.', STARTKEY => '', ENDKEY => ''} 2023-03-24 16:56:09,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 16:56:09,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,612 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,613 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A 2023-03-24 16:56:09,613 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A 2023-03-24 16:56:09,613 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:09,613 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName A 2023-03-24 16:56:09,615 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:09,616 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,616 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B 2023-03-24 16:56:09,616 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B 2023-03-24 16:56:09,616 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:09,617 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName B 2023-03-24 16:56:09,617 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:09,617 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,618 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C 2023-03-24 16:56:09,618 DEBUG [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C 2023-03-24 16:56:09,618 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 16:56:09,618 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa8f591d6f09d5e6894f62e442177704 columnFamilyName C 2023-03-24 16:56:09,618 INFO [StoreOpener-fa8f591d6f09d5e6894f62e442177704-1] regionserver.HStore(310): Store=fa8f591d6f09d5e6894f62e442177704/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 16:56:09,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 16:56:09,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fa8f591d6f09d5e6894f62e442177704; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70412078, jitterRate=0.04922172427177429}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 16:56:09,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:09,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., pid=61, masterSystemTime=1679676969608 2023-03-24 16:56:09,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,625 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,625 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676969625"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679676969625"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679676969625"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679676969625"}]},"ts":"1679676969625"} 2023-03-24 16:56:09,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-24 16:56:09,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 in 169 msec 2023-03-24 16:56:09,628 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-24 16:56:09,628 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, REOPEN/MOVE in 484 msec 2023-03-24 16:56:09,630 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-24 16:56:09,630 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-24 16:56:09,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-03-24 16:56:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-24 16:56:09,633 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,638 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,638 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,645 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,646 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,649 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,650 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,655 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,655 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,660 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,661 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,667 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,667 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,673 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,674 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,682 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,683 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,687 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,687 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:62195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 16:56:09,692 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 16:56:09,699 DEBUG [hconnection-0x23d90ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,699 DEBUG [hconnection-0x5c6814b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,700 DEBUG [hconnection-0x46ea5eb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,701 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46856, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:09,704 DEBUG [hconnection-0x7e5a62fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,708 DEBUG [hconnection-0x597cc4d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,712 DEBUG [hconnection-0x16364751-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,711 DEBUG [hconnection-0x40d4d87d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,710 DEBUG [hconnection-0x52002b54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,722 DEBUG [hconnection-0xeff2867-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,722 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46848, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,722 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46836, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,720 DEBUG [hconnection-0x25985f5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 16:56:09,724 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,724 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46884, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,724 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46868, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,726 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46898, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:09,726 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46912, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:09,729 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46900, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,733 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:09,733 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:09,733 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:09,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46928, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 16:56:09,733 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:09,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:09,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:09,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,740 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:09,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:09,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:09,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:09,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:09,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:09,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:09,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:09,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:09,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:09,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:09,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:09,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:09,743 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:09,744 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 16:56:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:09,745 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:09,745 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:09,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676969754, max timestamp=1679676969761Num uniques 859; 2023-03-24 16:56:09,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676969754, max timestamp=1679676969761Num uniques 859; 2023-03-24 16:56:09,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676969754, max timestamp=1679676969761Num uniques 859; 2023-03-24 16:56:09,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677029760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:09,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677029759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,778 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241fc6841e2e2044ed9ae600ae8e2e5789 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241fc6841e2e2044ed9ae600ae8e2e5789 2023-03-24 16:56:09,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677029780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677029781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677029781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,784 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/017741500ebb43f7b5609528a87d618e 2023-03-24 16:56:09,805 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/8caf44ac28f54fd3b16d7fc83b99f2cd 2023-03-24 16:56:09,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677029864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677029864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677029884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677029884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:09,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:09,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677029884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677030068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677030068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677030086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677030087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677030088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,222 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b3c4ec87346947d89e461555e6a5f0e6 2023-03-24 16:56:10,226 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/017741500ebb43f7b5609528a87d618e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e 2023-03-24 16:56:10,230 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e, entries=150, sequenceid=14, filesize=25.1 K 2023-03-24 16:56:10,231 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/8caf44ac28f54fd3b16d7fc83b99f2cd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd 2023-03-24 16:56:10,234 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 16:56:10,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b3c4ec87346947d89e461555e6a5f0e6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6 2023-03-24 16:56:10,238 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 16:56:10,239 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 495ms, sequenceid=14, compaction requested=false 2023-03-24 16:56:10,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:10,239 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,242 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f9a4746[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,246 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,250 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,251 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,254 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,258 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,258 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,259 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:10,259 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,359 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:10,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:10,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:10,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:10,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:10,362 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:10,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:10,362 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:10,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,364 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:10,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:10,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:10,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:10,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:10,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:10,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:10,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:10,367 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,367 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:10,367 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:10,367 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:10,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:10,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677030381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677030382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,386 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ac9ea17883d0425a8e1b8eb4b3fd0420 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac9ea17883d0425a8e1b8eb4b3fd0420 2023-03-24 16:56:10,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677030388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,390 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9e8b05dd98cd42b9ac6d5a51161d937f 2023-03-24 16:56:10,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677030390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677030392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,422 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/1b184e82317f4f2d9f5a3ac93db9f7c3 2023-03-24 16:56:10,432 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/34ec80c1a07e4ba984201bd79e009914 2023-03-24 16:56:10,435 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9e8b05dd98cd42b9ac6d5a51161d937f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f 2023-03-24 16:56:10,439 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f, entries=150, sequenceid=51, filesize=25.1 K 2023-03-24 16:56:10,439 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/1b184e82317f4f2d9f5a3ac93db9f7c3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3 2023-03-24 16:56:10,443 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 16:56:10,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/34ec80c1a07e4ba984201bd79e009914 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914 2023-03-24 16:56:10,449 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 16:56:10,450 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fa8f591d6f09d5e6894f62e442177704 in 83ms, sequenceid=51, compaction requested=false 2023-03-24 16:56:10,450 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:10,450 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,450 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,455 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ddb2876[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,460 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,462 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,466 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,466 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,466 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,466 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,466 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,471 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,474 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,474 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,474 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:10,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:10,498 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241dbf374476ca4b2aa9cd0a51e82a4b51 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241dbf374476ca4b2aa9cd0a51e82a4b51 2023-03-24 16:56:10,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676970506, max timestamp=1679676970514Num uniques 859; 2023-03-24 16:56:10,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:10,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676970506, max timestamp=1679676970514Num uniques 859; 2023-03-24 16:56:10,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:10,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676970506, max timestamp=1679676970514Num uniques 859; 2023-03-24 16:56:10,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677030538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677030540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,574 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:10,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:10,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:10,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:10,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:10,578 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:10,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:10,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:10,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,580 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,581 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:10,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:10,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:10,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:10,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:10,582 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d85b3b0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,585 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,586 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,587 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,591 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,591 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,598 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,599 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:10,599 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677030642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677030643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,699 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:10,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:10,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:10,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:10,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:10,703 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:10,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:10,703 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:10,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,705 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:10,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:10,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:10,708 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,708 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:10,708 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:10,708 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,710 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,710 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,711 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,711 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b0f895d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,713 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,715 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,716 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,717 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,717 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,721 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,721 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,721 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,723 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,724 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,725 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,727 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,727 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,728 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,728 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,728 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,728 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,728 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,728 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:10,828 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:10,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:10,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:10,832 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:10,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:10,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:10,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:10,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:10,837 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,837 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:10,837 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:10,837 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,839 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652410d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,841 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,841 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677030843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,845 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,845 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,850 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677030848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,853 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,853 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,853 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,855 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,855 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,855 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,856 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,858 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,861 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,861 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,861 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677030893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677030901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677030902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,905 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/d5b490f4e24c4c82bb9a4c02ad665bfe 2023-03-24 16:56:10,961 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:10,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:10,965 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:10,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:10,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:10,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:10,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:10,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:10,968 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:10,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:10,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:10,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:10,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,972 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428e4129[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:10,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:10,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,975 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:10,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,976 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:10,977 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:10,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:10,979 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:10,980 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:10,980 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:10,980 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:10,980 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:10,981 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,981 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,981 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:10,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:10,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,983 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:10,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:10,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:10,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:10,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:10,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:10,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:10,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:10,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,987 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:10,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:10,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:10,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:10,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:10,987 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:11,087 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:11,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:11,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:11,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:11,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:11,091 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,093 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,093 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:11,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,094 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,094 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:11,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:11,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:11,094 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:11,094 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,094 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:11,095 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,095 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:11,095 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:11,095 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:11,095 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:11,097 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,097 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:11,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25662fb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:11,099 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:11,099 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,101 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:11,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,101 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:11,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:11,106 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:11,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:11,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:11,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:11,106 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:11,108 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,108 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:11,108 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:11,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:11,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:11,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:11,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,114 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:11,114 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:11,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677031146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677031153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,214 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:11,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:11,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:11,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:11,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:11,217 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:11,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:11,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:11,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,219 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:11,219 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:11,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,220 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:11,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:11,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:11,221 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,221 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:11,221 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:11,221 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:11,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,225 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,225 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594568d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:11,226 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:11,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,228 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:11,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,229 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:11,232 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:11,235 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:11,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:11,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:11,235 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:11,235 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:11,237 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,237 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:11,237 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:11,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:11,243 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:11,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:11,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,243 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:11,243 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:11,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:11,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/5de278e14f6d4ecdae321f4b88306122 2023-03-24 16:56:11,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/4d7ccc3a16f44be49ebc1eb97d0e1af2 2023-03-24 16:56:11,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/d5b490f4e24c4c82bb9a4c02ad665bfe as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe 2023-03-24 16:56:11,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe, entries=150, sequenceid=62, filesize=25.1 K 2023-03-24 16:56:11,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/5de278e14f6d4ecdae321f4b88306122 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122 2023-03-24 16:56:11,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122, entries=150, sequenceid=62, filesize=11.6 K 2023-03-24 16:56:11,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/4d7ccc3a16f44be49ebc1eb97d0e1af2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2 2023-03-24 16:56:11,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2, entries=150, sequenceid=62, filesize=11.6 K 2023-03-24 16:56:11,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa8f591d6f09d5e6894f62e442177704 in 854ms, sequenceid=62, compaction requested=true 2023-03-24 16:56:11,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:11,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:11,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:11,339 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:11,339 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:11,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:11,340 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:11,340 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:11,340 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:11,341 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,341 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=75.4 K 2023-03-24 16:56:11,341 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:11,341 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,341 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=34.8 K 2023-03-24 16:56:11,341 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 017741500ebb43f7b5609528a87d618e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679676969739 2023-03-24 16:56:11,341 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 8caf44ac28f54fd3b16d7fc83b99f2cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679676969739 2023-03-24 16:56:11,341 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9e8b05dd98cd42b9ac6d5a51161d937f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679676969777 2023-03-24 16:56:11,342 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1b184e82317f4f2d9f5a3ac93db9f7c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679676969777 2023-03-24 16:56:11,342 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d5b490f4e24c4c82bb9a4c02ad665bfe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:11,342 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5de278e14f6d4ecdae321f4b88306122, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:11,344 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:11,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:11,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:11,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:11,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:11,348 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:11,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:11,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:11,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,349 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#557 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:11,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:11,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:11,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,350 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:11,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:11,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:11,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:11,356 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:11,356 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:11,356 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:11,356 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:11,372 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#558 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:11,460 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 16:56:11,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:11,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:11,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677031664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677031667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677031768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677031771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,775 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/ae3f76cee2e848478989a26497e53c86 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ae3f76cee2e848478989a26497e53c86 2023-03-24 16:56:11,779 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into ae3f76cee2e848478989a26497e53c86(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:11,779 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:11,779 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676971339; duration=0sec 2023-03-24 16:56:11,780 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:11,780 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:11,781 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:11,781 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:11,781 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:11,781 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=34.8 K 2023-03-24 16:56:11,781 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b3c4ec87346947d89e461555e6a5f0e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679676969739 2023-03-24 16:56:11,782 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 34ec80c1a07e4ba984201bd79e009914, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679676969777 2023-03-24 16:56:11,782 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4d7ccc3a16f44be49ebc1eb97d0e1af2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:11,788 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#560 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:11,791 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324aac9855c7fb141adbd0d63d3ac0823d7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324aac9855c7fb141adbd0d63d3ac0823d7 2023-03-24 16:56:11,800 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/fbf05197ae634907952869d5ac45ae09 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fbf05197ae634907952869d5ac45ae09 2023-03-24 16:56:11,801 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/a87f4571680c44cea0b3edfebc40b665 2023-03-24 16:56:11,804 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into fbf05197ae634907952869d5ac45ae09(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:11,804 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:11,804 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676971339; duration=0sec 2023-03-24 16:56:11,804 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:11,813 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/13b887cab1b148a9965bd554005e6d9e 2023-03-24 16:56:11,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677031898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677031904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677031912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677031970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677031975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,203 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/d9150d015dbf423da722850290c6366e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d9150d015dbf423da722850290c6366e 2023-03-24 16:56:12,210 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into d9150d015dbf423da722850290c6366e(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:12,210 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,210 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676971339; duration=0sec 2023-03-24 16:56:12,211 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:12,226 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b3c3746f649d4b7182e1e70c5f558eae 2023-03-24 16:56:12,229 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/a87f4571680c44cea0b3edfebc40b665 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665 2023-03-24 16:56:12,233 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665, entries=150, sequenceid=98, filesize=25.1 K 2023-03-24 16:56:12,234 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/13b887cab1b148a9965bd554005e6d9e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e 2023-03-24 16:56:12,237 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 16:56:12,237 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b3c3746f649d4b7182e1e70c5f558eae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae 2023-03-24 16:56:12,240 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 16:56:12,241 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for fa8f591d6f09d5e6894f62e442177704 in 885ms, sequenceid=98, compaction requested=false 2023-03-24 16:56:12,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@115a251b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,248 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,248 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:12,253 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:12,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:12,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:12,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:12,253 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:12,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:12,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:12,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:12,260 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,260 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:12,260 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:12,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:12,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,287 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242edaa9d171ee4606ad5e674fa0d5f041 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242edaa9d171ee4606ad5e674fa0d5f041 2023-03-24 16:56:12,291 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/caa2b840c1534410b3e65d505cc6f2f8 2023-03-24 16:56:12,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677032306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677032306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,360 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:12,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:12,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:12,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:12,364 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:12,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:12,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:12,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,367 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:12,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:12,378 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,378 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:12,378 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:12,378 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21edb910[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,383 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,383 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,385 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:12,389 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:12,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:12,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:12,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:12,389 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:12,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,393 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,395 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,397 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,397 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,397 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,397 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:12,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:12,398 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:12,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:12,398 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:12,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,398 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677032409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677032409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,498 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:12,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:12,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:12,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:12,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:12,503 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:12,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:12,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:12,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,505 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:12,508 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:12,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:12,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:12,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:12,510 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,513 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,513 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109d566c[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,514 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,514 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,518 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,521 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:12,524 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:12,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:12,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:12,524 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:12,524 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:12,528 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,528 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,528 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:12,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:12,540 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:12,540 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:12,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,540 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,539 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677032613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:12,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677032614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,640 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:12,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:12,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:12,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:12,643 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:12,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:12,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:12,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,643 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,647 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:12,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:12,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:12,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:12,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:12,649 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,651 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371a5acc[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,652 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,652 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,654 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,655 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,661 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,661 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:12,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:12,663 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:12,663 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:12,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,666 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,666 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,668 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:12,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:12,670 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,671 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,671 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,671 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,671 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:12,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,670 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,671 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:12,671 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:12,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/e1fd48f99f4045a39d0f8f2c5353a7ca 2023-03-24 16:56:12,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/ce1a95a9717f41a9b566fcf23db44ae3 2023-03-24 16:56:12,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/caa2b840c1534410b3e65d505cc6f2f8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8 2023-03-24 16:56:12,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8, entries=150, sequenceid=113, filesize=25.1 K 2023-03-24 16:56:12,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/e1fd48f99f4045a39d0f8f2c5353a7ca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca 2023-03-24 16:56:12,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca, entries=150, sequenceid=113, filesize=11.6 K 2023-03-24 16:56:12,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/ce1a95a9717f41a9b566fcf23db44ae3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3 2023-03-24 16:56:12,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3, entries=150, sequenceid=113, filesize=11.6 K 2023-03-24 16:56:12,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fa8f591d6f09d5e6894f62e442177704 in 483ms, sequenceid=113, compaction requested=true 2023-03-24 16:56:12,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:12,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:12,758 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:12,758 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:12,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:12,759 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:12,759 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:12,759 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:12,759 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:12,759 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,759 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,759 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d9150d015dbf423da722850290c6366e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=75.5 K 2023-03-24 16:56:12,759 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ae3f76cee2e848478989a26497e53c86, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=34.9 K 2023-03-24 16:56:12,760 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting d9150d015dbf423da722850290c6366e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:12,760 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a87f4571680c44cea0b3edfebc40b665, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679676970530 2023-03-24 16:56:12,761 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting caa2b840c1534410b3e65d505cc6f2f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:12,761 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ae3f76cee2e848478989a26497e53c86, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:12,762 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 13b887cab1b148a9965bd554005e6d9e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679676970530 2023-03-24 16:56:12,762 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e1fd48f99f4045a39d0f8f2c5353a7ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:12,771 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:12,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:12,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:12,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:12,776 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:12,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:12,776 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:12,777 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,777 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,777 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#567 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:12,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,778 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,778 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:12,780 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#566 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:12,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:12,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:12,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,784 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:12,792 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/007b1a97b06243249b1ec4ce671cfdd7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/007b1a97b06243249b1ec4ce671cfdd7 2023-03-24 16:56:12,795 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 007b1a97b06243249b1ec4ce671cfdd7(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:12,796 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,796 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676972758; duration=0sec 2023-03-24 16:56:12,796 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:12,796 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:12,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:12,796 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,796 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:12,796 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:12,811 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:12,797 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:12,811 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,811 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fbf05197ae634907952869d5ac45ae09, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=34.9 K 2023-03-24 16:56:12,811 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:56:12,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:12,811 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,811 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fbf05197ae634907952869d5ac45ae09, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679676970376 2023-03-24 16:56:12,812 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:12,812 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,812 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:12,812 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,812 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting b3c3746f649d4b7182e1e70c5f558eae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679676970530 2023-03-24 16:56:12,812 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ce1a95a9717f41a9b566fcf23db44ae3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:12,818 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#569 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:12,824 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a40bf82a850447898d29a6b2cd75203a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a40bf82a850447898d29a6b2cd75203a 2023-03-24 16:56:12,829 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/77a2ad3f34454f92adf78630e63dbf94 2023-03-24 16:56:12,829 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/879d48c0ce564bafbd69ba68afee3b59 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/879d48c0ce564bafbd69ba68afee3b59 2023-03-24 16:56:12,834 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 879d48c0ce564bafbd69ba68afee3b59(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:12,834 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,834 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676972758; duration=0sec 2023-03-24 16:56:12,834 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:12,838 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/be73d0a568964a2b847c2bbdc5f48d0d 2023-03-24 16:56:12,846 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/83d1fbfe6a7844ea9fa7d08c1461f8b4 2023-03-24 16:56:12,849 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/77a2ad3f34454f92adf78630e63dbf94 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94 2023-03-24 16:56:12,852 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94, entries=150, sequenceid=134, filesize=25.2 K 2023-03-24 16:56:12,852 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/be73d0a568964a2b847c2bbdc5f48d0d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d 2023-03-24 16:56:12,856 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d, entries=150, sequenceid=134, filesize=11.7 K 2023-03-24 16:56:12,856 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/83d1fbfe6a7844ea9fa7d08c1461f8b4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4 2023-03-24 16:56:12,860 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4, entries=150, sequenceid=134, filesize=11.7 K 2023-03-24 16:56:12,860 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 49ms, sequenceid=134, compaction requested=false 2023-03-24 16:56:12,860 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:12,860 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,863 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7728d3c9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,871 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:12,873 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:12,873 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:12,873 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:12,873 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:12,873 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:12,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,875 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,876 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,877 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:12,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:12,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:12,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,880 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,881 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:12,881 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:12,881 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,881 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,881 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:12,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:12,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:12,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:12,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:12,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:12,981 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:12,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:12,985 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:12,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,985 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:12,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:12,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,986 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:12,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:12,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:12,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:12,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:12,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:12,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:12,990 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,990 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:12,990 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:12,990 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:12,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,992 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,992 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:12,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:12,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc1c105[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:12,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:12,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:12,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:12,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,001 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,001 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,001 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,005 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677033004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676972996, max timestamp=1679676973005Num uniques 859; 2023-03-24 16:56:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676972996, max timestamp=1679676973005Num uniques 859; 2023-03-24 16:56:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676972996, max timestamp=1679676973005Num uniques 859; 2023-03-24 16:56:13,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:13,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,010 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,011 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,011 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677033059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677033108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,111 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,114 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,115 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,117 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,120 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4195ce27[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,124 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,127 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,127 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,127 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,129 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,129 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,130 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,134 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,134 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677033163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,234 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,239 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,241 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24bc20e3[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,249 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,251 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,253 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,253 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,253 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,255 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,256 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,261 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,261 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677033310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,361 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,366 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,369 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,369 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677033367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,371 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@153b0926[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,377 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,377 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,379 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,382 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,382 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,384 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,385 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249f00d0a6fabe41eba1d89ae62d44e489 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f00d0a6fabe41eba1d89ae62d44e489 2023-03-24 16:56:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,386 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,391 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9a9da088804b4c279f9459f3f21e0df3 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,392 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,392 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,392 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,392 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,493 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,496 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,496 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,496 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,496 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,497 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,500 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,500 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,506 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,506 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,507 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,507 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20472b6d[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,512 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,515 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,515 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,516 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,522 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,523 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,523 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,523 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677033615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,623 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,627 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,630 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/425960ee6ba74ccd817d8b812621a616 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/425960ee6ba74ccd817d8b812621a616 2023-03-24 16:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,631 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,632 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,632 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,633 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,633 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,633 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,635 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into 425960ee6ba74ccd817d8b812621a616(size=25.3 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:13,635 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:13,635 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676972758; duration=0sec 2023-03-24 16:56:13,635 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:13,635 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,636 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,636 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcc2db4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,638 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,639 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,639 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,639 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,642 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,644 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,644 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,644 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,650 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,651 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,651 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,652 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677033673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,752 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,755 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,757 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,758 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,759 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,759 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,759 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,759 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,759 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,762 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,762 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,762 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,762 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab2b796[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,764 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,765 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,765 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,766 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,767 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,767 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,769 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,769 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,769 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,769 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,769 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,771 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,773 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,775 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,775 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,776 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,776 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,776 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/6b8e503dc53b4fcd9d7b8da8152ee63f 2023-03-24 16:56:13,876 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:13,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:13,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:13,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:13,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:13,879 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:13,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:13,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:13,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,881 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,881 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,882 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:13,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:13,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:13,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:13,883 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,883 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:13,883 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:13,883 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:13,883 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:13,885 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470dfddf[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:13,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:13,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,891 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:13,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,892 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,893 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:13,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:13,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:13,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:13,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:13,896 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:13,896 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:13,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:13,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,899 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:13,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:13,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,900 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677033901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:13,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,903 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,904 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:13,904 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:13,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:13,905 DEBUG [Thread-4125] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4124 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:13,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:13,905 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:13,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:13,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:13,905 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:13,906 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:13,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:13,906 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:13,906 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:13,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677033913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,915 DEBUG [Thread-4127] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:13,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:13,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677033922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:13,924 DEBUG [Thread-4131] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:14,006 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,010 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,012 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,012 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,014 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,014 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,015 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:14,015 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e88ba10[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,018 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,019 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,020 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,020 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,020 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,020 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,020 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,022 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,025 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,027 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,028 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,028 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,028 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,029 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,032 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,032 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,032 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,032 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,033 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,033 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677034118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,132 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,135 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,136 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,137 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,137 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,137 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,137 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,138 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,139 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,139 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,139 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:14,139 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,141 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70879df8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,142 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,142 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,145 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,145 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,147 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,149 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,149 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,151 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,151 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,151 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,151 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,153 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,154 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,155 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,155 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,155 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677034180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/e9ff342e82684b77b3faa274219d9a06 2023-03-24 16:56:14,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9a9da088804b4c279f9459f3f21e0df3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3 2023-03-24 16:56:14,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3, entries=350, sequenceid=146, filesize=52.6 K 2023-03-24 16:56:14,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/6b8e503dc53b4fcd9d7b8da8152ee63f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f 2023-03-24 16:56:14,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f, entries=150, sequenceid=146, filesize=11.8 K 2023-03-24 16:56:14,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/e9ff342e82684b77b3faa274219d9a06 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06 2023-03-24 16:56:14,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06, entries=150, sequenceid=146, filesize=11.8 K 2023-03-24 16:56:14,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa8f591d6f09d5e6894f62e442177704 in 1258ms, sequenceid=146, compaction requested=true 2023-03-24 16:56:14,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:14,227 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:14,227 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:14,227 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:14,227 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:14,227 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36114 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:14,228 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,228 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:14,228 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/425960ee6ba74ccd817d8b812621a616, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=103.2 K 2023-03-24 16:56:14,228 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,228 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/007b1a97b06243249b1ec4ce671cfdd7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=35.3 K 2023-03-24 16:56:14,228 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 425960ee6ba74ccd817d8b812621a616, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:14,228 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 007b1a97b06243249b1ec4ce671cfdd7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:14,229 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 77a2ad3f34454f92adf78630e63dbf94, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679676972297 2023-03-24 16:56:14,229 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting be73d0a568964a2b847c2bbdc5f48d0d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679676972297 2023-03-24 16:56:14,229 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 9a9da088804b4c279f9459f3f21e0df3, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679676972934 2023-03-24 16:56:14,229 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 6b8e503dc53b4fcd9d7b8da8152ee63f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679676972934 2023-03-24 16:56:14,234 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#576 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:14,245 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#575 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:14,255 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,259 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,261 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,261 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,262 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,262 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,268 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,268 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:14,269 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:14,284 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248fc7314e85904642bb9a93b7e8c79858 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fc7314e85904642bb9a93b7e8c79858 2023-03-24 16:56:14,290 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c44c58d762ec4eec8e040cba7dbf0efc 2023-03-24 16:56:14,301 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3d8b1d4b503e43f2a7d1973f31b611ba 2023-03-24 16:56:14,309 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/442539240bf145d087b61645589f13f2 2023-03-24 16:56:14,312 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c44c58d762ec4eec8e040cba7dbf0efc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc 2023-03-24 16:56:14,315 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc, entries=150, sequenceid=183, filesize=25.3 K 2023-03-24 16:56:14,316 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3d8b1d4b503e43f2a7d1973f31b611ba as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba 2023-03-24 16:56:14,319 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba, entries=150, sequenceid=183, filesize=11.8 K 2023-03-24 16:56:14,319 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/442539240bf145d087b61645589f13f2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2 2023-03-24 16:56:14,322 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2, entries=150, sequenceid=183, filesize=11.8 K 2023-03-24 16:56:14,323 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 55ms, sequenceid=183, compaction requested=true 2023-03-24 16:56:14,323 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,323 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,323 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f168a79[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,333 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,334 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,334 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,338 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,341 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,341 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,341 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,341 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,341 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,342 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,343 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,345 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,348 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,348 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,349 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,349 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,349 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,449 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,453 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,453 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,454 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,455 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,456 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,458 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475d6141[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,460 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,461 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,461 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,464 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,467 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,467 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,467 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,468 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,469 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,473 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,473 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,573 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,577 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,577 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,579 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,581 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,585 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c1e1b80[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,587 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,588 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,588 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,588 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,589 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,589 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,592 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,593 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,593 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,594 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,595 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,596 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,599 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,599 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,600 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,600 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,600 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,656 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/f323698c620e4d318683f425fe5f32bb as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f323698c620e4d318683f425fe5f32bb 2023-03-24 16:56:14,665 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into f323698c620e4d318683f425fe5f32bb(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:14,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,666 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676974227; duration=0sec 2023-03-24 16:56:14,666 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:14,666 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:56:14,667 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48150 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:56:14,667 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:14,667 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,667 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/879d48c0ce564bafbd69ba68afee3b59, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=47.0 K 2023-03-24 16:56:14,667 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 879d48c0ce564bafbd69ba68afee3b59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679676971662 2023-03-24 16:56:14,668 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 83d1fbfe6a7844ea9fa7d08c1461f8b4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679676972297 2023-03-24 16:56:14,668 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e9ff342e82684b77b3faa274219d9a06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679676972934 2023-03-24 16:56:14,669 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 442539240bf145d087b61645589f13f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679676973047 2023-03-24 16:56:14,694 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#580 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:14,700 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,708 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,708 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,709 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,710 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,710 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,712 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/4e50e3ebcce74ce9a94aab1dca26d325 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4e50e3ebcce74ce9a94aab1dca26d325 2023-03-24 16:56:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,712 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,712 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,713 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,713 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,714 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,714 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,715 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,716 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 4e50e3ebcce74ce9a94aab1dca26d325(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:14,716 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,716 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=12, startTime=1679676974227; duration=0sec 2023-03-24 16:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,716 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ba35a6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,717 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,718 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,718 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,718 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,720 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,721 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,721 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,723 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,723 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,723 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,725 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,726 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,730 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,735 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,735 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,736 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,736 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,836 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,839 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,839 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,840 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,842 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,842 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,843 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,843 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,844 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,844 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,845 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,845 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,846 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b24308a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,848 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,849 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,849 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,850 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,850 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,851 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,851 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,854 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,854 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,854 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,857 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,858 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,860 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:14,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,860 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,861 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,861 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,861 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,861 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,861 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,861 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,961 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:14,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:14,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:14,965 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:14,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:14,965 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:14,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,966 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,967 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,967 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:14,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:14,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:14,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:14,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:14,969 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:14,971 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675f17c6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:14,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:14,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:14,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,976 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:14,978 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:14,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:14,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:14,979 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:14,979 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:14,980 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,980 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,980 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:14,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:14,981 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,982 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:14,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:14,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:14,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:14,984 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,984 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:14,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:14,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:14,985 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:14,985 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:14,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:14,986 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:14,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:14,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,085 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,088 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,088 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,089 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,091 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,091 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,092 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,101 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,101 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,101 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,101 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:15,102 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:15,103 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:15,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb1d2b6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:15,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:15,106 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,107 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,107 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:15,109 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,110 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/cd4c86c6191343cfa95f6072585934c6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/cd4c86c6191343cfa95f6072585934c6 2023-03-24 16:56:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:15,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:15,113 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:15,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:15,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:15,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:15,113 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:15,114 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into cd4c86c6191343cfa95f6072585934c6(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:15,114 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:15,115 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676974226; duration=0sec 2023-03-24 16:56:15,115 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:15,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,116 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,122 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:15,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:15,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,123 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,123 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:15,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:15,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:15,203 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245f402c6a682541eabee4a15c587facc4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245f402c6a682541eabee4a15c587facc4 2023-03-24 16:56:15,211 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/b2427f1e0f754c01af84689a281a8d88 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676975208, max timestamp=1679676975217Num uniques 859; 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676975208, max timestamp=1679676975217Num uniques 859; 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676975208, max timestamp=1679676975217Num uniques 859; 2023-03-24 16:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:15,223 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/4df62e41c368426ea95462bb336aa362 2023-03-24 16:56:15,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,227 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,227 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,228 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,229 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,229 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,230 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,231 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,231 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,232 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:15,232 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:15,234 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,234 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:15,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a67d963[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:15,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:15,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:15,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,240 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:15,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:15,243 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:15,243 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:15,243 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:15,243 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:15,243 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:15,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,244 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,246 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:15,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:15,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:15,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,251 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:15,250 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,251 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:15,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677035257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677035258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,351 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,354 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,354 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,356 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,358 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,358 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,358 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:15,358 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,360 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677035361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677035361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:15,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602d18c5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:15,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:15,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,366 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:15,367 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:15,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,367 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:15,369 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:15,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:15,371 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:15,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:15,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:15,372 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:15,372 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:15,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,373 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,374 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,376 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,378 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:15,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:15,378 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:15,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,379 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:15,379 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:15,479 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,482 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,482 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,482 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,483 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,483 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,483 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,484 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,486 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,486 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,486 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:15,486 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:15,486 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:15,488 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:15,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd3c221[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:15,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:15,490 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,491 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:15,491 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,491 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:15,492 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,492 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:15,493 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,493 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:15,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:15,495 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:15,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:15,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:15,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:15,495 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:15,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,497 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:15,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:15,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,502 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,502 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,503 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:15,503 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:15,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,503 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46868 deadline: 1679677035564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677035564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,603 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,606 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,606 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,606 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,606 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,607 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,612 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,612 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,612 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,612 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,614 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,614 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,614 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:15,614 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:15,616 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,616 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,616 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab29b0f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:15,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:15,617 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,618 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:15,619 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,619 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:15,620 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:15,622 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:15,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:15,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:15,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:15,622 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:15,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:15,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,626 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:15,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:15,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:15,630 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:15,630 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:15,630 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:15,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/366ae493198f4a489be0346ea2a73099 2023-03-24 16:56:15,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/b2427f1e0f754c01af84689a281a8d88 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88 2023-03-24 16:56:15,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88, entries=300, sequenceid=202, filesize=45.8 K 2023-03-24 16:56:15,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/4df62e41c368426ea95462bb336aa362 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362 2023-03-24 16:56:15,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362, entries=150, sequenceid=202, filesize=11.8 K 2023-03-24 16:56:15,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/366ae493198f4a489be0346ea2a73099 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099 2023-03-24 16:56:15,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099, entries=150, sequenceid=202, filesize=11.8 K 2023-03-24 16:56:15,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 480ms, sequenceid=202, compaction requested=true 2023-03-24 16:56:15,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:15,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:15,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:15,662 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:15,662 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:15,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:15,663 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:15,663 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:15,663 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,663 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/cd4c86c6191343cfa95f6072585934c6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=96.7 K 2023-03-24 16:56:15,663 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting cd4c86c6191343cfa95f6072585934c6, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679676972934 2023-03-24 16:56:15,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,664 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f323698c620e4d318683f425fe5f32bb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=35.6 K 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c44c58d762ec4eec8e040cba7dbf0efc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679676973047 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f323698c620e4d318683f425fe5f32bb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679676972934 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b2427f1e0f754c01af84689a281a8d88, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679676975159 2023-03-24 16:56:15,664 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3d8b1d4b503e43f2a7d1973f31b611ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679676973047 2023-03-24 16:56:15,665 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4df62e41c368426ea95462bb336aa362, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679676975168 2023-03-24 16:56:15,671 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#585 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:15,689 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#584 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:15,730 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:15,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:15,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:15,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:15,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:15,734 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:15,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:15,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:15,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,734 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,736 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:15,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:15,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,736 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:15,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:15,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:15,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:15,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:15,738 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:15,738 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:15,748 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:15,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:15,761 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248951189c748440af9cf8419b12c05190 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248951189c748440af9cf8419b12c05190 2023-03-24 16:56:15,766 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/4e2919d209614ccd8ef9b67420243309 2023-03-24 16:56:15,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:15,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:15,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677035881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46872 deadline: 1679677035883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677035984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:15,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:15,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677035987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,098 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/1c799e5ef20d40578c44efeb51831bda as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1c799e5ef20d40578c44efeb51831bda 2023-03-24 16:56:16,103 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 1c799e5ef20d40578c44efeb51831bda(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:16,103 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:16,103 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676975662; duration=0sec 2023-03-24 16:56:16,103 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:16,103 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 16:56:16,104 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 16:56:16,104 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 16:56:16,105 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. because compaction request was cancelled 2023-03-24 16:56:16,175 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d427831229854c77ae34a10e9b4146f8 2023-03-24 16:56:16,183 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5 2023-03-24 16:56:16,185 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/4e2919d209614ccd8ef9b67420243309 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309 2023-03-24 16:56:16,191 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309, entries=150, sequenceid=239, filesize=25.3 K 2023-03-24 16:56:16,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677036189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677036190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,192 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d427831229854c77ae34a10e9b4146f8 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8 2023-03-24 16:56:16,195 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8, entries=150, sequenceid=239, filesize=11.8 K 2023-03-24 16:56:16,196 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5 2023-03-24 16:56:16,202 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5, entries=150, sequenceid=239, filesize=11.8 K 2023-03-24 16:56:16,202 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fa8f591d6f09d5e6894f62e442177704 in 464ms, sequenceid=239, compaction requested=true 2023-03-24 16:56:16,202 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:16,202 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:16,204 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:16,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,204 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2ae1fb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:16,206 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:16,207 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,208 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,209 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:16,209 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:16,210 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:16,213 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:16,214 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:16,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:16,216 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:16,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:16,217 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,218 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:16,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:16,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:16,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:16,221 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:16,221 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:16,222 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:16,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:16,222 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:16,222 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,322 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:16,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:16,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:16,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:16,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:16,325 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:16,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,326 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:16,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,328 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:16,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:16,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:16,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:16,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:16,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:16,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:16,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:16,330 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:16,330 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:16,330 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:16,330 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:16,331 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:16,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:16,511 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/a86cd49609b945818263b532d38b5048 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a86cd49609b945818263b532d38b5048 2023-03-24 16:56:16,514 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into a86cd49609b945818263b532d38b5048(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:16,515 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:16,515 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676975662; duration=0sec 2023-03-24 16:56:16,515 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:16,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676976519, max timestamp=1679676976527Num uniques 859; 2023-03-24 16:56:16,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:16,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676976519, max timestamp=1679676976527Num uniques 859; 2023-03-24 16:56:16,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676976519, max timestamp=1679676976527Num uniques 859; 2023-03-24 16:56:16,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:16,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:16,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676976550, max timestamp=1679676976560Num uniques 1018; 2023-03-24 16:56:16,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:16,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676976550, max timestamp=1679676976560Num uniques 1018; 2023-03-24 16:56:16,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676976550, max timestamp=1679676976560Num uniques 1018; 2023-03-24 16:56:16,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:16,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:16,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677036591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677036591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677036695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677036695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,748 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032422eaab69f6124b70980377ef26a6c832 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422eaab69f6124b70980377ef26a6c832 2023-03-24 16:56:16,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677036898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:16,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:16,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677036899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,154 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7760604c8f4b41f39543a10c0381495a 2023-03-24 16:56:17,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677037203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677037205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,566 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/2693a6fcb73c49b3978eac9b81340fa0 2023-03-24 16:56:17,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677037708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677037712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677037913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,916 DEBUG [Thread-4125] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:17,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677037922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,925 DEBUG [Thread-4127] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:17,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:17,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677037948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,951 DEBUG [Thread-4131] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:17,978 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/6b7e7fc3e2ad4eab8784e49749dc87a1 2023-03-24 16:56:17,982 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7760604c8f4b41f39543a10c0381495a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a 2023-03-24 16:56:17,985 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a, entries=150, sequenceid=250, filesize=25.3 K 2023-03-24 16:56:17,985 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/2693a6fcb73c49b3978eac9b81340fa0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0 2023-03-24 16:56:17,988 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0, entries=150, sequenceid=250, filesize=11.8 K 2023-03-24 16:56:17,989 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/6b7e7fc3e2ad4eab8784e49749dc87a1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1 2023-03-24 16:56:17,992 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1, entries=150, sequenceid=250, filesize=11.8 K 2023-03-24 16:56:17,993 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 1662ms, sequenceid=250, compaction requested=true 2023-03-24 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:17,994 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6492bcb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:17,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:17,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:17,999 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:18,001 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:18,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:18,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:18,002 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:18,002 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:18,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:18,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:18,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:18,005 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:18,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:18,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:18,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:18,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:18,014 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:18,014 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:18,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:18,114 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:18,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:18,118 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:18,119 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,119 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:18,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:18,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:18,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:18,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:18,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:18,121 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:18,122 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:18,537 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303243aef2d4176fb407ca21f4430a13d0b75 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243aef2d4176fb407ca21f4430a13d0b75 2023-03-24 16:56:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:18,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:18,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:18,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677038733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677038733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:18,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677038836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677038837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,943 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/dfff822ccedf4ee3bc43eefe6adad152 2023-03-24 16:56:18,954 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/c22cff962528453785cff726abae81c6 2023-03-24 16:56:18,966 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/1d624438899a45df99ffb55e10b4e693 2023-03-24 16:56:18,970 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/dfff822ccedf4ee3bc43eefe6adad152 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152 2023-03-24 16:56:18,975 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152, entries=150, sequenceid=305, filesize=25.4 K 2023-03-24 16:56:18,976 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/c22cff962528453785cff726abae81c6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6 2023-03-24 16:56:18,981 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6, entries=150, sequenceid=305, filesize=11.9 K 2023-03-24 16:56:18,981 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/1d624438899a45df99ffb55e10b4e693 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693 2023-03-24 16:56:18,984 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693, entries=150, sequenceid=305, filesize=11.9 K 2023-03-24 16:56:18,985 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=46.96 KB/48090 for fa8f591d6f09d5e6894f62e442177704 in 864ms, sequenceid=305, compaction requested=true 2023-03-24 16:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:18,987 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ea23e9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:18,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:18,989 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,996 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:18,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:18,997 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,997 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:18,998 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:19,000 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:19,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:19,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:19,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:19,001 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:19,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,002 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,004 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,006 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,007 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:19,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:19,007 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:19,007 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:19,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:19,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:19,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:19,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677039069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676979058, max timestamp=1679676979070Num uniques 859; 2023-03-24 16:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676979058, max timestamp=1679676979070Num uniques 859; 2023-03-24 16:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676979058, max timestamp=1679676979070Num uniques 859; 2023-03-24 16:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:19,108 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:19,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:19,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:19,111 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,113 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,113 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,114 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:19,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:19,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:19,117 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:19,118 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,118 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49aea6f8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:19,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:19,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,121 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:19,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:19,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:19,126 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:19,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:19,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:19,126 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:19,126 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:19,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,129 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677039129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,133 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:19,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,134 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:19,134 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:19,133 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,134 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677039176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,234 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:19,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677039233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:19,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:19,237 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:19,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:19,237 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:19,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,238 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,239 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,239 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:19,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:19,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:19,243 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1052b30f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:19,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:19,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,247 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:19,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:19,248 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:19,252 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:19,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:19,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:19,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:19,252 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:19,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,254 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,255 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,260 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:19,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:19,260 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,261 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:19,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,261 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:19,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:19,361 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:19,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:19,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:19,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:19,365 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:19,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:19,365 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:19,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,366 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,368 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,368 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:19,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:19,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:19,369 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,369 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:19,369 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:19,370 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:19,371 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:19,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0ac5bd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,374 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:19,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:19,379 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:19,382 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:19,382 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:19,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677039379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,383 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:19,384 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,387 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:19,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:19,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:19,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:19,388 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,388 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:19,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,388 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:19,388 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:19,388 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:19,388 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:19,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677039435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,455 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245920713ff10a4240af49184e771280b2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245920713ff10a4240af49184e771280b2 2023-03-24 16:56:19,459 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/e21d5bdba75d4b0dad4bee907b404e20 2023-03-24 16:56:19,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/f525653dd3df4403aaccfb6bd84f831e 2023-03-24 16:56:19,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/f757e0f462744cffa680b23217cbf933 2023-03-24 16:56:19,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/e21d5bdba75d4b0dad4bee907b404e20 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20 2023-03-24 16:56:19,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20, entries=150, sequenceid=316, filesize=25.4 K 2023-03-24 16:56:19,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/f525653dd3df4403aaccfb6bd84f831e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e 2023-03-24 16:56:19,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e, entries=150, sequenceid=316, filesize=11.9 K 2023-03-24 16:56:19,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/f757e0f462744cffa680b23217cbf933 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933 2023-03-24 16:56:19,488 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:19,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:19,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:19,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933, entries=150, sequenceid=316, filesize=11.9 K 2023-03-24 16:56:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:19,492 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:19,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa8f591d6f09d5e6894f62e442177704 in 450ms, sequenceid=316, compaction requested=true 2023-03-24 16:56:19,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:19,493 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:19,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:19,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:19,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:19,493 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:19,495 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130190 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:19,495 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:19,495 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,495 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a86cd49609b945818263b532d38b5048, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=127.1 K 2023-03-24 16:56:19,496 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting a86cd49609b945818263b532d38b5048, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679676975168 2023-03-24 16:56:19,496 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4e2919d209614ccd8ef9b67420243309, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679676975243 2023-03-24 16:56:19,497 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60890 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:19,497 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:19,497 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,497 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1c799e5ef20d40578c44efeb51831bda, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=59.5 K 2023-03-24 16:56:19,497 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1c799e5ef20d40578c44efeb51831bda, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679676975168 2023-03-24 16:56:19,498 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:19,498 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7760604c8f4b41f39543a10c0381495a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1679676975879 2023-03-24 16:56:19,498 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,498 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d427831229854c77ae34a10e9b4146f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679676975243 2023-03-24 16:56:19,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:19,499 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting dfff822ccedf4ee3bc43eefe6adad152, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679676976585 2023-03-24 16:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:19,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:19,499 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2693a6fcb73c49b3978eac9b81340fa0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1679676975879 2023-03-24 16:56:19,499 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e21d5bdba75d4b0dad4bee907b404e20, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:19,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:19,500 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting c22cff962528453785cff726abae81c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679676976585 2023-03-24 16:56:19,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:19,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:19,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:19,501 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f525653dd3df4403aaccfb6bd84f831e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:19,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:19,501 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,501 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:19,501 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:19,501 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:19,502 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:19,518 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#599 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:19,525 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#598 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:19,531 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/2ea50b3d81e741cd8255480b03a73ddd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2ea50b3d81e741cd8255480b03a73ddd 2023-03-24 16:56:19,535 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 2ea50b3d81e741cd8255480b03a73ddd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:19,535 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:19,535 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=11, startTime=1679676979493; duration=0sec 2023-03-24 16:56:19,535 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:19,535 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:56:19,537 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72858 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:56:19,537 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:19,537 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:19,537 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4e50e3ebcce74ce9a94aab1dca26d325, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=71.2 K 2023-03-24 16:56:19,538 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4e50e3ebcce74ce9a94aab1dca26d325, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679676973047 2023-03-24 16:56:19,538 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 366ae493198f4a489be0346ea2a73099, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679676975168 2023-03-24 16:56:19,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting fdd9ba6c76f24746ae3bcb9a14c1e4b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679676975243 2023-03-24 16:56:19,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 6b7e7fc3e2ad4eab8784e49749dc87a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1679676975879 2023-03-24 16:56:19,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1d624438899a45df99ffb55e10b4e693, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679676976585 2023-03-24 16:56:19,539 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f757e0f462744cffa680b23217cbf933, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:19,543 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324cd88711926de4e6eb4a99d3de7a20f32 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cd88711926de4e6eb4a99d3de7a20f32 2023-03-24 16:56:19,559 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#601 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:19,561 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/1ebdad2ba8cc4f6b8e69ad3132edc587 2023-03-24 16:56:19,575 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/4c56df83945649959551fd82ea5ae3ae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4c56df83945649959551fd82ea5ae3ae 2023-03-24 16:56:19,579 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 4c56df83945649959551fd82ea5ae3ae(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:19,579 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:19,579 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=10, startTime=1679676979493; duration=0sec 2023-03-24 16:56:19,579 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:19,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:19,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:19,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677039704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677039740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677039807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:19,960 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c3d66e47dd2241d099cb6d7472e32d7e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c3d66e47dd2241d099cb6d7472e32d7e 2023-03-24 16:56:19,964 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into c3d66e47dd2241d099cb6d7472e32d7e(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:19,964 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:19,964 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=11, startTime=1679676979493; duration=0sec 2023-03-24 16:56:19,964 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:19,982 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/26a3e367180c474091df2652bf831745 2023-03-24 16:56:19,994 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/337ec43eb9cd4edb84fec7b16085fa62 2023-03-24 16:56:19,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/1ebdad2ba8cc4f6b8e69ad3132edc587 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587 2023-03-24 16:56:20,000 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587, entries=150, sequenceid=352, filesize=25.4 K 2023-03-24 16:56:20,001 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/26a3e367180c474091df2652bf831745 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745 2023-03-24 16:56:20,004 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745, entries=150, sequenceid=352, filesize=11.9 K 2023-03-24 16:56:20,005 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/337ec43eb9cd4edb84fec7b16085fa62 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62 2023-03-24 16:56:20,009 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62, entries=150, sequenceid=352, filesize=11.9 K 2023-03-24 16:56:20,010 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for fa8f591d6f09d5e6894f62e442177704 in 509ms, sequenceid=352, compaction requested=false 2023-03-24 16:56:20,010 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:20,010 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,012 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,012 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,013 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 16:56:20,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,016 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c940de[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:20,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:20,017 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,017 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,018 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,028 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,033 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,033 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,036 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,036 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,036 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,043 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,055 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,056 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,056 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,056 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,057 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,064 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d413cc7a4eff4583bdcc6361495f3150 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d413cc7a4eff4583bdcc6361495f3150 2023-03-24 16:56:20,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:20,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677040088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,157 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,160 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,160 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,161 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,162 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,164 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556ca279[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,166 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,168 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,169 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,171 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,171 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,173 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,175 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,177 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,178 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:20,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677040192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:20,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677040243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,278 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,281 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,283 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,283 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,285 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,287 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a3b910[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,289 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,290 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,290 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,291 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,294 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,294 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,297 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,297 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,297 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,304 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,305 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,304 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,305 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:20,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677040396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,405 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,412 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,414 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,416 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,416 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,416 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,416 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,418 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,418 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,419 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590da994[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,424 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,426 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,430 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,430 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,430 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,433 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,436 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,436 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,437 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,437 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,480 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/8dfdac030e6c49278063355d22b5675c 2023-03-24 16:56:20,537 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,540 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,540 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,547 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,549 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,549 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,549 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,549 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,549 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a61dddb[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,553 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,554 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,555 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,556 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,560 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,560 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,560 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,566 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,567 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,568 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,568 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,668 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,672 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,672 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,673 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,673 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,674 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,675 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,675 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,675 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,676 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,677 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,678 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aaed784[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,679 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,681 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,682 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,686 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,686 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,686 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,687 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,695 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,695 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677040698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,796 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,799 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,799 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,801 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,801 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,802 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,802 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,802 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,803 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,803 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,804 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,804 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7088cf8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,806 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,807 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,807 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,807 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,810 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,810 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,812 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,812 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,812 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,815 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,817 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,817 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,817 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,817 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,818 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,818 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,818 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:20,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3c97f1d151264b3d86f50805d1f95646 2023-03-24 16:56:20,918 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:20,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:20,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:20,921 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:20,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:20,921 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:20,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,922 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,923 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,923 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:20,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:20,924 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:20,925 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,925 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:20,925 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:20,925 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:20,928 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,928 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337dd484[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:20,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:20,930 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,931 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:20,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,932 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,932 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:20,937 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:20,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:20,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:20,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:20,937 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:20,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,938 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:20,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,939 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,941 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:20,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:20,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:20,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:20,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:20,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:20,944 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:20,945 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:20,944 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:20,945 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:20,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:20,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:20,945 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:20,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:21,045 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:21,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:21,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:21,052 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:21,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:21,052 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:21,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,053 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,054 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,054 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:21,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:21,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:21,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:21,057 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:21,058 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,059 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47064835[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:21,060 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:21,061 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,061 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:21,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:21,062 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:21,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,062 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:21,064 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:21,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,064 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:21,066 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:21,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:21,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:21,067 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:21,067 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:21,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,068 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,069 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,070 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,073 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:21,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:21,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,073 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:21,073 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,073 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,173 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:21,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:21,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:21,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:21,177 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:21,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:21,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:21,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,182 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,182 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:21,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:21,184 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,184 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:21,184 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:21,184 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:21,184 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:21,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,186 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,186 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b28a5a6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:21,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:21,188 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,189 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:21,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:21,189 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:21,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,190 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:21,192 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:21,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,192 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:21,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:21,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:21,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:21,195 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:21,195 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:21,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,197 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,197 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,198 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,201 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:21,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:21,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:21,201 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,202 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:21,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,202 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:21,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,202 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677041203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:21,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677041249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a8c6c35dc37b4267b984a5e8056cf06f 2023-03-24 16:56:21,302 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:21,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:21,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/8dfdac030e6c49278063355d22b5675c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c 2023-03-24 16:56:21,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:21,305 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:21,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:21,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:21,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c, entries=150, sequenceid=367, filesize=25.4 K 2023-03-24 16:56:21,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,307 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,307 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:21,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3c97f1d151264b3d86f50805d1f95646 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646 2023-03-24 16:56:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:21,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:21,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:21,309 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:21,309 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,309 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:21,309 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:21,310 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:21,310 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:21,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646, entries=150, sequenceid=367, filesize=11.9 K 2023-03-24 16:56:21,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a8c6c35dc37b4267b984a5e8056cf06f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f 2023-03-24 16:56:21,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f, entries=150, sequenceid=367, filesize=11.9 K 2023-03-24 16:56:21,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fa8f591d6f09d5e6894f62e442177704 in 1299ms, sequenceid=367, compaction requested=true 2023-03-24 16:56:21,315 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:21,316 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,316 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:21,316 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:21,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,317 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78718 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:21,317 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:21,317 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,317 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c3d66e47dd2241d099cb6d7472e32d7e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=76.9 K 2023-03-24 16:56:21,317 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:21,317 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,317 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c3d66e47dd2241d099cb6d7472e32d7e, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:21,317 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,317 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2ea50b3d81e741cd8255480b03a73ddd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.3 K 2023-03-24 16:56:21,318 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1ebdad2ba8cc4f6b8e69ad3132edc587, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676979113 2023-03-24 16:56:21,318 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 2ea50b3d81e741cd8255480b03a73ddd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,318 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 8dfdac030e6c49278063355d22b5675c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:21,318 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 26a3e367180c474091df2652bf831745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676979113 2023-03-24 16:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,319 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3c97f1d151264b3d86f50805d1f95646, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38848977[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:21,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:21,319 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,320 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:21,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,321 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:21,322 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,327 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:21,328 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:21,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:21,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:21,328 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:21,328 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:21,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,330 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,333 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#607 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:21,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,337 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,337 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,337 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:21,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:21,337 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,438 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:21,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:21,441 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:21,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:21,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:21,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:21,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:21,444 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:21,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:21,445 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:21,445 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:21,445 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:21,461 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032428be81ae79794efaaf3618762caad8eb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428be81ae79794efaaf3618762caad8eb 2023-03-24 16:56:21,758 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3b5ba07803a046c2b9937f2338c2ff62 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3b5ba07803a046c2b9937f2338c2ff62 2023-03-24 16:56:21,764 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 3b5ba07803a046c2b9937f2338c2ff62(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:21,764 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:21,764 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676981316; duration=0sec 2023-03-24 16:56:21,764 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:21,764 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:21,765 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:21,765 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:21,766 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,766 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4c56df83945649959551fd82ea5ae3ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.2 K 2023-03-24 16:56:21,766 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 4c56df83945649959551fd82ea5ae3ae, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1679676978724 2023-03-24 16:56:21,766 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 337ec43eb9cd4edb84fec7b16085fa62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679676979113 2023-03-24 16:56:21,766 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a8c6c35dc37b4267b984a5e8056cf06f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:21,776 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:21,786 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/cf89e6b9cf824afd9056ca4f3b3d97ca as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cf89e6b9cf824afd9056ca4f3b3d97ca 2023-03-24 16:56:21,790 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into cf89e6b9cf824afd9056ca4f3b3d97ca(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:21,790 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:21,790 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676981316; duration=0sec 2023-03-24 16:56:21,790 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:21,866 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/5654510b990e4c2ca0640d4d25dc3cfa 2023-03-24 16:56:21,875 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/ff48f5bbf4ae4ee1a9decfc3934aaa13 2023-03-24 16:56:21,885 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a380eaedf87e473ea67fe5f279da7807 2023-03-24 16:56:21,888 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/5654510b990e4c2ca0640d4d25dc3cfa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa 2023-03-24 16:56:21,892 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa, entries=150, sequenceid=387, filesize=25.4 K 2023-03-24 16:56:21,893 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/ff48f5bbf4ae4ee1a9decfc3934aaa13 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13 2023-03-24 16:56:21,896 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13, entries=150, sequenceid=387, filesize=11.9 K 2023-03-24 16:56:21,897 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a380eaedf87e473ea67fe5f279da7807 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807 2023-03-24 16:56:21,900 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807, entries=150, sequenceid=387, filesize=11.9 K 2023-03-24 16:56:21,900 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 455ms, sequenceid=387, compaction requested=false 2023-03-24 16:56:21,900 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:21,901 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:21,902 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:21,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6500b317[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:21,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:21,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,905 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,906 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:21,908 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,908 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:21,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:21,911 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:21,911 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:21,911 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:21,911 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:21,911 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:21,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:21,913 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:21,913 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,914 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:21,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:21,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:21,917 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:21,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:21,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:21,917 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:21,917 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:21,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,018 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,021 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,022 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,022 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,023 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,024 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,024 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,025 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:22,025 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,029 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10320862[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,031 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,033 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,033 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,034 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,037 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,037 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,037 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,040 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,042 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,044 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,044 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,045 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,044 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:22,045 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,145 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,148 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,149 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,150 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,150 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,150 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,150 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:22,153 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,155 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,155 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e098a57[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,157 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,159 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,159 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,159 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,160 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,163 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,163 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,163 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,165 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,167 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,168 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,169 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/4bad458673964441b66512d70b3a41b9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4bad458673964441b66512d70b3a41b9 2023-03-24 16:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:22,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,171 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,171 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,171 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,173 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into 4bad458673964441b66512d70b3a41b9(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:22,173 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:22,173 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676981316; duration=0sec 2023-03-24 16:56:22,173 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:22,271 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,276 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,279 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,279 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:22,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:22,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:22,288 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325a61d0[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,294 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,297 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,298 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,301 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,301 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,304 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,305 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,306 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,309 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,309 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,310 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,310 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,310 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,320 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032401ef2bfa5e734a9c84c2ba5d3bbb204f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032401ef2bfa5e734a9c84c2ba5d3bbb204f 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676982311, max timestamp=1679676982338Num uniques 859; 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676982311, max timestamp=1679676982338Num uniques 859; 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676982311, max timestamp=1679676982338Num uniques 859; 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:22,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:22,342 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=402, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/45d78a197eed4909b5cc7b8d9ef00382 2023-03-24 16:56:22,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d71dde924b514395a1247e0dd4967e83 2023-03-24 16:56:22,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677042388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,410 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,413 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,414 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,416 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,417 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,423 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,425 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5222f487[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,431 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,432 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,435 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,435 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,438 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,440 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,441 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,443 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,444 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,443 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,444 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,444 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:22,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677042491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,544 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,548 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,548 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,552 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,553 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,553 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,553 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:22,553 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,553 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,554 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,555 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525387b9[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,558 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,558 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,559 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,563 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,564 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,564 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,566 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,567 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,580 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,583 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,583 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,583 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,583 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,683 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,687 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,688 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,689 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,689 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:22,691 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:22,693 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30eaf496[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:22,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:22,695 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:22,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677042694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,700 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:22,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:22,700 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:22,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,700 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:22,701 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:22,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,701 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:22,704 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:22,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:22,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:22,704 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:22,704 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:22,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:22,705 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,706 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,707 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,720 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:22,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:22,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:22,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:22,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,720 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:22,720 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:22,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:22,720 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:22,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:22,721 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/5f16f299e7b840819ed25c0062a6f532 2023-03-24 16:56:22,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/45d78a197eed4909b5cc7b8d9ef00382 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382 2023-03-24 16:56:22,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382, entries=300, sequenceid=402, filesize=46.1 K 2023-03-24 16:56:22,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d71dde924b514395a1247e0dd4967e83 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83 2023-03-24 16:56:22,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83, entries=100, sequenceid=402, filesize=9.5 K 2023-03-24 16:56:22,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/5f16f299e7b840819ed25c0062a6f532 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532 2023-03-24 16:56:22,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532, entries=100, sequenceid=402, filesize=9.5 K 2023-03-24 16:56:22,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 526ms, sequenceid=402, compaction requested=true 2023-03-24 16:56:22,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:22,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:22,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:22,807 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:22,807 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:22,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:56:22,808 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99940 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:22,809 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:22,809 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,809 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4bad458673964441b66512d70b3a41b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=97.6 K 2023-03-24 16:56:22,809 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:22,809 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:22,809 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,809 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3b5ba07803a046c2b9937f2338c2ff62, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=34.0 K 2023-03-24 16:56:22,809 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 4bad458673964441b66512d70b3a41b9, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:22,810 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 5654510b990e4c2ca0640d4d25dc3cfa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1679676980073 2023-03-24 16:56:22,810 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3b5ba07803a046c2b9937f2338c2ff62, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:22,810 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 45d78a197eed4909b5cc7b8d9ef00382, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676982253 2023-03-24 16:56:22,810 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ff48f5bbf4ae4ee1a9decfc3934aaa13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1679676980073 2023-03-24 16:56:22,810 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d71dde924b514395a1247e0dd4967e83, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676982253 2023-03-24 16:56:22,816 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#617 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:22,821 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:22,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:22,832 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:22,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:22,832 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:22,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,833 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,835 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:22,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:22,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,835 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:22,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:22,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:22,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:22,845 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#616 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:22,847 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:22,847 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:22,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:22,850 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/7a1d193c77c546c9b72cce32b6e7fca3 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/7a1d193c77c546c9b72cce32b6e7fca3 2023-03-24 16:56:22,854 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 7a1d193c77c546c9b72cce32b6e7fca3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:22,854 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:22,854 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676982807; duration=0sec 2023-03-24 16:56:22,854 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:22,854 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:22,855 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:22,855 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:22,855 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:22,855 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cf89e6b9cf824afd9056ca4f3b3d97ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=33.9 K 2023-03-24 16:56:22,856 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cf89e6b9cf824afd9056ca4f3b3d97ca, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679676979692 2023-03-24 16:56:22,857 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a380eaedf87e473ea67fe5f279da7807, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1679676980073 2023-03-24 16:56:22,857 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5f16f299e7b840819ed25c0062a6f532, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676982253 2023-03-24 16:56:22,863 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#619 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:22,863 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f7222389cb5a406487c4e2bf1045e87f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7222389cb5a406487c4e2bf1045e87f 2023-03-24 16:56:23,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:23,001 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:23,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:23,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677043014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677043117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:23,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677043257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,260 DEBUG [Thread-4129] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:23,270 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/e4665b2c8d3443e5bd5729e3bc794c63 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e4665b2c8d3443e5bd5729e3bc794c63 2023-03-24 16:56:23,272 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7e64d3e3230f491f9230775fe1cab63f 2023-03-24 16:56:23,276 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/5d831db8097f4269b7bf0a76b4570d52 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5d831db8097f4269b7bf0a76b4570d52 2023-03-24 16:56:23,278 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into e4665b2c8d3443e5bd5729e3bc794c63(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:23,278 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:23,278 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676982807; duration=0sec 2023-03-24 16:56:23,278 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:23,282 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 5d831db8097f4269b7bf0a76b4570d52(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:23,282 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:23,282 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676982807; duration=0sec 2023-03-24 16:56:23,282 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:23,285 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/48d5562dc36f4118875adafac7c43d4e 2023-03-24 16:56:23,301 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/59819dedd001415cacdac633bf44c86f 2023-03-24 16:56:23,305 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7e64d3e3230f491f9230775fe1cab63f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f 2023-03-24 16:56:23,313 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f, entries=150, sequenceid=439, filesize=25.4 K 2023-03-24 16:56:23,314 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/48d5562dc36f4118875adafac7c43d4e as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e 2023-03-24 16:56:23,320 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e, entries=150, sequenceid=439, filesize=11.9 K 2023-03-24 16:56:23,321 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/59819dedd001415cacdac633bf44c86f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f 2023-03-24 16:56:23,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:23,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677043321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,324 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f, entries=150, sequenceid=439, filesize=11.9 K 2023-03-24 16:56:23,325 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fa8f591d6f09d5e6894f62e442177704 in 478ms, sequenceid=439, compaction requested=false 2023-03-24 16:56:23,325 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:23,325 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:23,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:23,326 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:23,327 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515f47ba[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:23,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:23,330 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,331 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:23,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,332 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:23,333 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:23,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:23,336 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:23,336 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:23,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,347 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:23,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,347 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,349 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:23,351 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:23,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:23,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:23,351 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,352 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,352 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:23,352 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:23,452 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:23,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:23,456 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,457 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,457 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:23,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,458 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:23,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:23,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:23,460 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:23,460 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,481 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324273c58a71d8e4708b265163720fd922b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324273c58a71d8e4708b265163720fd922b 2023-03-24 16:56:23,488 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/b496c06bee144236a7de10a4e0183b28 2023-03-24 16:56:23,501 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/ba4fa1a837d64d0a92c695c8f7a83df7 2023-03-24 16:56:23,510 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/79db461905644002b3825eb59f623590 2023-03-24 16:56:23,513 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/b496c06bee144236a7de10a4e0183b28 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28 2023-03-24 16:56:23,516 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28, entries=150, sequenceid=452, filesize=25.4 K 2023-03-24 16:56:23,517 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/ba4fa1a837d64d0a92c695c8f7a83df7 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7 2023-03-24 16:56:23,520 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7, entries=150, sequenceid=452, filesize=11.9 K 2023-03-24 16:56:23,521 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/79db461905644002b3825eb59f623590 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590 2023-03-24 16:56:23,525 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590, entries=150, sequenceid=452, filesize=11.9 K 2023-03-24 16:56:23,526 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 66ms, sequenceid=452, compaction requested=true 2023-03-24 16:56:23,526 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:23,526 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:23,534 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141c17da[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:23,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:23,536 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,537 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:23,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:23,539 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:23,542 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:23,542 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:23,542 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:23,542 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:23,542 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:23,544 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:23,544 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,544 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:23,545 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,547 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,550 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:23,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:23,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:23,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:23,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:23,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:23,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:23,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,551 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:23,551 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:23,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:23,551 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:23,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,651 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:23,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:23,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:23,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:23,655 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:23,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:23,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:23,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:23,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,655 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,657 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:23,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:23,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:23,657 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:23,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:23,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:23,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:23,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:23,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:23,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:23,658 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:23,658 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:23,659 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:23,659 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:23,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:23,673 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e3be999fac7041c88c06752ff0b61bef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3be999fac7041c88c06752ff0b61bef 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676983699, max timestamp=1679676983711Num uniques 859; 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676983699, max timestamp=1679676983711Num uniques 859; 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676983699, max timestamp=1679676983711Num uniques 859; 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:23,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:23,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676983752, max timestamp=1679676983762Num uniques 1018; 2023-03-24 16:56:23,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:23,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676983752, max timestamp=1679676983762Num uniques 1018; 2023-03-24 16:56:23,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679676983752, max timestamp=1679676983762Num uniques 1018; 2023-03-24 16:56:23,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:23,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 16:56:23,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676983820, max timestamp=1679676983830Num uniques 1027; 2023-03-24 16:56:23,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676983820, max timestamp=1679676983830Num uniques 1027; 2023-03-24 16:56:23,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679676983820, max timestamp=1679676983830Num uniques 1027; 2023-03-24 16:56:23,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:56:23,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:56:23,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 16:56:23,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676983864, max timestamp=1679676983881Num uniques 1036; 2023-03-24 16:56:23,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 16:56:23,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676983864, max timestamp=1679676983881Num uniques 1036; 2023-03-24 16:56:23,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 16:56:23,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679676983864, max timestamp=1679676983881Num uniques 1036; 2023-03-24 16:56:23,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 16:56:23,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676983918, max timestamp=1679676983936Num uniques 1045; 2023-03-24 16:56:23,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:56:23,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676983918, max timestamp=1679676983936Num uniques 1045; 2023-03-24 16:56:23,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:56:23,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679676983918, max timestamp=1679676983936Num uniques 1045; 2023-03-24 16:56:23,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-24 16:56:24,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676983989, max timestamp=1679676984003Num uniques 1054; 2023-03-24 16:56:24,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:56:24,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676983989, max timestamp=1679676984003Num uniques 1054; 2023-03-24 16:56:24,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:56:24,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679676983989, max timestamp=1679676984003Num uniques 1054; 2023-03-24 16:56:24,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 16:56:24,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676984042, max timestamp=1679676984055Num uniques 1013; 2023-03-24 16:56:24,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 16:56:24,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676984042, max timestamp=1679676984055Num uniques 1013; 2023-03-24 16:56:24,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679676984042, max timestamp=1679676984055Num uniques 1013; 2023-03-24 16:56:24,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 16:56:24,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 16:56:24,082 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/f7f005b006464fe697622d2e6711ce93 2023-03-24 16:56:24,093 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/74556d2dba24412389a2039b91155faf 2023-03-24 16:56:24,103 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/e92246120b7c4ded8bd68fd195fdfa31 2023-03-24 16:56:24,106 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/f7f005b006464fe697622d2e6711ce93 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93 2023-03-24 16:56:24,118 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93, entries=200, sequenceid=459, filesize=32.3 K 2023-03-24 16:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/74556d2dba24412389a2039b91155faf as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf 2023-03-24 16:56:24,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676984099, max timestamp=1679676984118Num uniques 1022; 2023-03-24 16:56:24,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-24 16:56:24,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676984099, max timestamp=1679676984118Num uniques 1022; 2023-03-24 16:56:24,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-24 16:56:24,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679676984099, max timestamp=1679676984118Num uniques 1022; 2023-03-24 16:56:24,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-24 16:56:24,124 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf, entries=100, sequenceid=459, filesize=9.5 K 2023-03-24 16:56:24,125 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/e92246120b7c4ded8bd68fd195fdfa31 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31 2023-03-24 16:56:24,128 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31, entries=100, sequenceid=459, filesize=9.5 K 2023-03-24 16:56:24,129 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=33.54 KB/34350 for fa8f591d6f09d5e6894f62e442177704 in 470ms, sequenceid=459, compaction requested=true 2023-03-24 16:56:24,129 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:24,129 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,129 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,131 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3c34af[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,134 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,135 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:24,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=53.67 KB heapSize=144.70 KB 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:24,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,143 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,143 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,145 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,145 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,145 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,146 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,148 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,152 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,153 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,154 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,154 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,154 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,185 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303246fe0bae4a5c64a7c8d4cd6ad26200ad6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246fe0bae4a5c64a7c8d4cd6ad26200ad6 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676984222, max timestamp=1679676984239Num uniques 859; 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676984222, max timestamp=1679676984239Num uniques 859; 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676984222, max timestamp=1679676984239Num uniques 859; 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:24,254 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,257 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,257 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,258 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,258 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,259 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,259 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,264 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,264 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,264 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:24,264 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,266 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,266 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,266 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d15353a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,268 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,269 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,271 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,274 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,274 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,275 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,275 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,275 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,277 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,284 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,284 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,284 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,284 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677044307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,384 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,388 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,389 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,390 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:24,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,402 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165cc7b5[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,403 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,405 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,405 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,407 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,410 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,410 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,410 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,411 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,411 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,412 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,412 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,413 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677044411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,415 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,416 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,415 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,416 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,416 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,516 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,519 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,520 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,521 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,521 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:24,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:24,523 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620aa063[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,529 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,531 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,534 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,534 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,534 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,536 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,537 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,538 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,541 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,541 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,541 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,541 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,597 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=604, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/3b1b9f308df842ee9e189ffada3c55c6 2023-03-24 16:56:24,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3413521963594b9c94169f31c75dbca2 2023-03-24 16:56:24,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677044617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/cc645a3a944b45ec89ef8d88a5e18d26 2023-03-24 16:56:24,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/3b1b9f308df842ee9e189ffada3c55c6 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6 2023-03-24 16:56:24,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6, entries=200, sequenceid=604, filesize=32.3 K 2023-03-24 16:56:24,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/3413521963594b9c94169f31c75dbca2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2 2023-03-24 16:56:24,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2, entries=150, sequenceid=604, filesize=11.9 K 2023-03-24 16:56:24,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/cc645a3a944b45ec89ef8d88a5e18d26 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26 2023-03-24 16:56:24,641 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26, entries=150, sequenceid=604, filesize=11.9 K 2023-03-24 16:56:24,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~144.66 KB/148128, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 503ms, sequenceid=604, compaction requested=true 2023-03-24 16:56:24,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:24,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:24,644 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:24,644 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:24,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:24,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,645 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,645 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,646 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145090 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:24,646 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:24,646 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59270 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:24,646 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,646 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:24,646 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e4665b2c8d3443e5bd5729e3bc794c63, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=141.7 K 2023-03-24 16:56:24,646 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,646 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/7a1d193c77c546c9b72cce32b6e7fca3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=57.9 K 2023-03-24 16:56:24,646 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,647 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,647 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting e4665b2c8d3443e5bd5729e3bc794c63, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676980087 2023-03-24 16:56:24,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,647 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,647 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7a1d193c77c546c9b72cce32b6e7fca3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676980087 2023-03-24 16:56:24,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,648 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 7e64d3e3230f491f9230775fe1cab63f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1679676982374 2023-03-24 16:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,648 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 48d5562dc36f4118875adafac7c43d4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1679676982374 2023-03-24 16:56:24,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:24,648 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ba4fa1a837d64d0a92c695c8f7a83df7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676983002 2023-03-24 16:56:24,648 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b496c06bee144236a7de10a4e0183b28, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676983002 2023-03-24 16:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,648 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 74556d2dba24412389a2039b91155faf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1679676983650 2023-03-24 16:56:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,649 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting f7f005b006464fe697622d2e6711ce93, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1679676983639 2023-03-24 16:56:24,649 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3413521963594b9c94169f31c75dbca2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1679676984118 2023-03-24 16:56:24,649 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 3b1b9f308df842ee9e189ffada3c55c6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1679676984118 2023-03-24 16:56:24,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,649 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,650 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:24,650 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:24,661 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#632 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:24,673 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#633 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:24,685 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324edea393225b9475ca1e7a535fc4a9bdc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324edea393225b9475ca1e7a535fc4a9bdc 2023-03-24 16:56:24,702 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=641, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/3b8c0a753e044459b59c2ed0246fded5 2023-03-24 16:56:24,714 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/fc00f885f9e9491fbff236aeed066515 2023-03-24 16:56:24,722 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a03b2132c8544654b70ac72a3202d06c 2023-03-24 16:56:24,726 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/3b8c0a753e044459b59c2ed0246fded5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5 2023-03-24 16:56:24,730 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5, entries=150, sequenceid=641, filesize=25.4 K 2023-03-24 16:56:24,731 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/fc00f885f9e9491fbff236aeed066515 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515 2023-03-24 16:56:24,734 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515, entries=150, sequenceid=641, filesize=11.9 K 2023-03-24 16:56:24,734 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/a03b2132c8544654b70ac72a3202d06c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c 2023-03-24 16:56:24,741 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c, entries=150, sequenceid=641, filesize=11.9 K 2023-03-24 16:56:24,742 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 92ms, sequenceid=641, compaction requested=true 2023-03-24 16:56:24,742 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:24,742 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,743 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,743 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,744 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592869c1[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,745 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,746 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,746 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,747 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,747 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,749 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,750 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,752 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,752 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,752 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,753 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,754 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,755 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,757 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,758 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,758 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,758 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,758 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,858 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,862 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,862 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,863 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,863 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,869 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,871 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,871 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,871 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:24,871 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:24,873 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d003b3b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:24,875 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:24,875 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,876 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,876 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,876 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:24,877 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,877 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,878 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:24,880 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:24,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:24,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:24,880 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:24,880 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:24,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:24,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,883 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,883 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:24,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,885 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:24,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,886 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,887 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:24,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:24,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:24,887 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:24,887 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:24,887 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:24,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:24,987 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:24,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:24,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:24,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:24,992 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:24,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:24,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:24,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,992 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,993 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:24,993 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:24,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,994 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:24,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:24,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:24,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:24,995 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:24,995 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:24,996 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 16:56:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:24,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:24,996 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:25,009 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249171f0f7f6a149deaec0eb9bc3bc6ce3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249171f0f7f6a149deaec0eb9bc3bc6ce3 2023-03-24 16:56:25,016 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=652, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7a5d3397647d4d1f903895f08c20171c 2023-03-24 16:56:25,025 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b64518d757ee44d5bd3580af5cf67ea2 2023-03-24 16:56:25,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676985058, max timestamp=1679676985068Num uniques 859; 2023-03-24 16:56:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676985058, max timestamp=1679676985068Num uniques 859; 2023-03-24 16:56:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676985058, max timestamp=1679676985068Num uniques 859; 2023-03-24 16:56:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:25,087 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/fa1d540486144d4285cd0e0a397415ae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fa1d540486144d4285cd0e0a397415ae 2023-03-24 16:56:25,091 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into fa1d540486144d4285cd0e0a397415ae(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:25,091 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:25,091 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=11, startTime=1679676984644; duration=0sec 2023-03-24 16:56:25,092 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:25,092 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 16:56:25,093 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71422 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 16:56:25,093 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:25,093 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:25,093 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5d831db8097f4269b7bf0a76b4570d52, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=69.7 K 2023-03-24 16:56:25,094 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5d831db8097f4269b7bf0a76b4570d52, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679676980087 2023-03-24 16:56:25,094 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 59819dedd001415cacdac633bf44c86f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1679676982374 2023-03-24 16:56:25,094 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 79db461905644002b3825eb59f623590, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679676983002 2023-03-24 16:56:25,095 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e92246120b7c4ded8bd68fd195fdfa31, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1679676983650 2023-03-24 16:56:25,095 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting cc645a3a944b45ec89ef8d88a5e18d26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1679676984118 2023-03-24 16:56:25,095 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a03b2132c8544654b70ac72a3202d06c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679676984288 2023-03-24 16:56:25,104 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#639 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:25,111 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/5c6e2e93d7c449acb18927f884bfb309 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5c6e2e93d7c449acb18927f884bfb309 2023-03-24 16:56:25,113 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/04ecb305fd6d40eebed7fbeab0d668df as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/04ecb305fd6d40eebed7fbeab0d668df 2023-03-24 16:56:25,115 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into 5c6e2e93d7c449acb18927f884bfb309(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:25,115 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:25,115 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=11, startTime=1679676984644; duration=0sec 2023-03-24 16:56:25,115 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:25,117 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 04ecb305fd6d40eebed7fbeab0d668df(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:25,117 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:25,117 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=10, startTime=1679676984644; duration=0sec 2023-03-24 16:56:25,117 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:25,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:25,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 523 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677045137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677045240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,437 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b18e909124dc402bb81eda80aca0b4dd 2023-03-24 16:56:25,440 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/7a5d3397647d4d1f903895f08c20171c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c 2023-03-24 16:56:25,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:25,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 527 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677045443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,443 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c, entries=250, sequenceid=652, filesize=39.2 K 2023-03-24 16:56:25,444 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b64518d757ee44d5bd3580af5cf67ea2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2 2023-03-24 16:56:25,447 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2, entries=150, sequenceid=652, filesize=11.9 K 2023-03-24 16:56:25,448 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b18e909124dc402bb81eda80aca0b4dd as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd 2023-03-24 16:56:25,451 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd, entries=150, sequenceid=652, filesize=11.9 K 2023-03-24 16:56:25,451 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 455ms, sequenceid=652, compaction requested=true 2023-03-24 16:56:25,451 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:25,451 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:25,454 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,455 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d912c41[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:25,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:25,456 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,463 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:25,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,464 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:25,465 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:25,468 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:25,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:25,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:25,468 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:25,468 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:25,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,469 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:25,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:25,470 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:25,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,473 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:25,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:25,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:25,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:25,473 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:25,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:25,474 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:25,474 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:25,474 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:25,574 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:25,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:25,578 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:25,578 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,579 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:25,584 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:25,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:25,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,584 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:25,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:25,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:25,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:25,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:25,587 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:25,587 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:25,587 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:25,601 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324be66b10fcba8453a9410a05574e585f2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324be66b10fcba8453a9410a05574e585f2 2023-03-24 16:56:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:25,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:25,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:25,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 536 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677045766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 538 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677045870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,014 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=692, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/542f5b45aabc41b2b7a99b7f503f4877 2023-03-24 16:56:26,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:26,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677046073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:26,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677046378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,422 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/72fbc3de227a4dd196d762aca4957a82 2023-03-24 16:56:26,837 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/39118033f6d045f18dbdf54872630c75 2023-03-24 16:56:26,841 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/542f5b45aabc41b2b7a99b7f503f4877 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877 2023-03-24 16:56:26,844 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877, entries=150, sequenceid=692, filesize=25.4 K 2023-03-24 16:56:26,845 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/72fbc3de227a4dd196d762aca4957a82 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82 2023-03-24 16:56:26,848 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82, entries=150, sequenceid=692, filesize=11.9 K 2023-03-24 16:56:26,848 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/39118033f6d045f18dbdf54872630c75 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75 2023-03-24 16:56:26,851 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75, entries=150, sequenceid=692, filesize=11.9 K 2023-03-24 16:56:26,852 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fa8f591d6f09d5e6894f62e442177704 in 1265ms, sequenceid=692, compaction requested=true 2023-03-24 16:56:26,852 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:26,852 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:26,853 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2f9a28[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:26,855 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:26,855 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,856 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:26,857 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,857 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:26,858 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:26,861 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:26,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:26,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:26,861 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:26,861 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:26,864 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,864 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,864 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:26,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:26,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:26,865 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,866 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:26,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:26,868 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:26,868 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,869 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:26,869 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:26,869 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:26,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:26,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:26,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:26,900 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032498ba4a1bc8f84847ad6c9a19bd2835ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032498ba4a1bc8f84847ad6c9a19bd2835ab 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676986905, max timestamp=1679676986949Num uniques 859; 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676986905, max timestamp=1679676986949Num uniques 859; 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679676986905, max timestamp=1679676986949Num uniques 859; 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:26,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 16:56:26,969 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:26,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:26,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:26,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:26,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:26,973 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:26,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:26,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:26,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:26,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,973 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:26,974 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:26,974 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:26,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:26,975 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:26,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:26,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:26,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:26,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:26,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:26,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:26,977 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:26,985 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10538ccd[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:26,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:26,987 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,988 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:26,993 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:26,996 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:26,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:26,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:26,996 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:26,996 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:26,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,997 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:26,999 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,000 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,003 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,004 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,004 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,004 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 578 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677047019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,104 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,108 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,108 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,109 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,110 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,110 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,111 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:27,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,112 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,112 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,112 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:27,112 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,114 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,114 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,114 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,114 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c46a41f[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,116 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,117 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,118 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,119 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,122 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,122 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,122 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,124 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,125 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 580 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677047124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,128 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,128 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,128 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,128 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,228 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,232 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,233 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,233 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,234 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:27,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:27,235 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,238 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,239 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680178f2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,240 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,241 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,241 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,241 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,242 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,245 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,245 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,245 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,246 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,246 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,246 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,248 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,251 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,252 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,252 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,252 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,252 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:27,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46872 deadline: 1679677047287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,291 DEBUG [Thread-4129] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., hostname=jenkins-hbase4.apache.org,33315,1679676819242, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 16:56:27,306 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=703, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/43edaec48d9648ccb15cbfd8dfbbafb2 2023-03-24 16:56:27,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:27,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 582 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677047329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,352 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,355 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,355 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,356 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,357 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,359 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,359 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,359 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:27,359 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,361 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e094100[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,363 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,364 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,365 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,371 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,373 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,374 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,374 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,375 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,375 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,375 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,375 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,377 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,379 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,380 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,380 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,381 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,381 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,381 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,481 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,484 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,484 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,485 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,486 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,486 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,487 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:27,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:27,488 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,491 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@123093e8[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,493 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,494 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,495 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,495 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,496 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,496 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,496 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,496 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,498 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,499 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,499 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,501 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,501 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,501 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,501 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,503 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,509 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,510 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,510 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,610 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,614 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,614 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,616 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,616 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,618 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,618 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,618 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:27,618 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,620 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3fabed[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,622 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,623 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,623 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,625 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,627 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,627 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,628 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,628 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,629 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,630 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,631 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,634 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,633 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,634 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,633 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,634 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 584 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677047633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b20c7118409943e3b4f139cf9d6a47f2 2023-03-24 16:56:27,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/89e72baa66274f58ba62e995f46a9b50 2023-03-24 16:56:27,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/43edaec48d9648ccb15cbfd8dfbbafb2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2 2023-03-24 16:56:27,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2, entries=200, sequenceid=703, filesize=32.3 K 2023-03-24 16:56:27,734 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:27,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b20c7118409943e3b4f139cf9d6a47f2 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2 2023-03-24 16:56:27,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:27,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:27,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:27,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:27,738 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:27,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,738 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:27,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:27,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2, entries=150, sequenceid=703, filesize=11.9 K 2023-03-24 16:56:27,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,739 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/89e72baa66274f58ba62e995f46a9b50 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50 2023-03-24 16:56:27,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,740 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,740 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50, entries=150, sequenceid=703, filesize=11.9 K 2023-03-24 16:56:27,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:27,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa8f591d6f09d5e6894f62e442177704 in 859ms, sequenceid=703, compaction requested=true 2023-03-24 16:56:27,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:27,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:27,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:27,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:27,746 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:27,746 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 16:56:27,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:27,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:27,747 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:27,747 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:27,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:27,748 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:27,748 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:27,749 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:27,749 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,749 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fa1d540486144d4285cd0e0a397415ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=60.4 K 2023-03-24 16:56:27,749 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 152308 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 16:56:27,749 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:27,749 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting fa1d540486144d4285cd0e0a397415ae, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1679676984118 2023-03-24 16:56:27,749 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,749 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5c6e2e93d7c449acb18927f884bfb309, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=148.7 K 2023-03-24 16:56:27,750 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting fc00f885f9e9491fbff236aeed066515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679676984288 2023-03-24 16:56:27,750 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 5c6e2e93d7c449acb18927f884bfb309, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1679676984118 2023-03-24 16:56:27,750 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b64518d757ee44d5bd3580af5cf67ea2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1679676984944 2023-03-24 16:56:27,751 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 3b8c0a753e044459b59c2ed0246fded5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679676984288 2023-03-24 16:56:27,751 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 72fbc3de227a4dd196d762aca4957a82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679676985115 2023-03-24 16:56:27,751 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 7a5d3397647d4d1f903895f08c20171c, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1679676984944 2023-03-24 16:56:27,752 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b20c7118409943e3b4f139cf9d6a47f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:27,752 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 542f5b45aabc41b2b7a99b7f503f4877, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679676985115 2023-03-24 16:56:27,757 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 43edaec48d9648ccb15cbfd8dfbbafb2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:27,770 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#647 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:27,800 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ae51b8caa9bf45f3b0aeb05983c760b4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ae51b8caa9bf45f3b0aeb05983c760b4 2023-03-24 16:56:27,802 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#648 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:27,806 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/f5c7e8b746494c7e8b6071ccd65a1f9f as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f5c7e8b746494c7e8b6071ccd65a1f9f 2023-03-24 16:56:27,811 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into f5c7e8b746494c7e8b6071ccd65a1f9f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:27,811 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:27,811 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=11, startTime=1679676987746; duration=0sec 2023-03-24 16:56:27,811 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:27,811 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 16:56:27,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 16:56:27,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:27,812 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,812 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/04ecb305fd6d40eebed7fbeab0d668df, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=48.5 K 2023-03-24 16:56:27,812 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 04ecb305fd6d40eebed7fbeab0d668df, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679676984288 2023-03-24 16:56:27,813 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting b18e909124dc402bb81eda80aca0b4dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1679676984944 2023-03-24 16:56:27,813 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 39118033f6d045f18dbdf54872630c75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679676985115 2023-03-24 16:56:27,814 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=739, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/050940459e1d42699c7ab71d5bbd28c9 2023-03-24 16:56:27,814 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 89e72baa66274f58ba62e995f46a9b50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:27,829 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#650 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:27,837 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d44a0333ac1a48fe8401b4bebdeb09e9 2023-03-24 16:56:27,851 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/ace17bbbe83e48d7b63a738663153e1d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ace17bbbe83e48d7b63a738663153e1d 2023-03-24 16:56:27,855 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into ace17bbbe83e48d7b63a738663153e1d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:27,855 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:27,856 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=12, startTime=1679676987746; duration=0sec 2023-03-24 16:56:27,856 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:27,866 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/804cb47b1469451ab5314572ef1eaae9 2023-03-24 16:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/050940459e1d42699c7ab71d5bbd28c9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9 2023-03-24 16:56:27,879 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9, entries=150, sequenceid=739, filesize=25.4 K 2023-03-24 16:56:27,879 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/d44a0333ac1a48fe8401b4bebdeb09e9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9 2023-03-24 16:56:27,885 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9, entries=150, sequenceid=739, filesize=11.9 K 2023-03-24 16:56:27,886 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/804cb47b1469451ab5314572ef1eaae9 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9 2023-03-24 16:56:27,890 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9, entries=150, sequenceid=739, filesize=11.9 K 2023-03-24 16:56:27,891 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 144ms, sequenceid=739, compaction requested=false 2023-03-24 16:56:27,891 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:27,891 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:27,892 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:27,894 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,895 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181a8a90[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:27,897 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:27,897 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,898 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,898 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:27,901 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:27,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:27,904 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:27,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:27,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:27,904 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:27,904 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:27,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,915 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,915 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:27,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,917 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:27,919 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:27,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:27,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:27,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:27,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:27,919 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:27,920 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:27,920 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:27,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:27,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:28,020 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,021 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032483617b8424604b35856e54e9e1fbc4e4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032483617b8424604b35856e54e9e1fbc4e4 2023-03-24 16:56:28,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,024 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,024 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,025 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,026 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,026 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,027 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676988019, max timestamp=1679676988029Num uniques 859; 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676988019, max timestamp=1679676988029Num uniques 859; 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679676988019, max timestamp=1679676988029Num uniques 859; 2023-03-24 16:56:28,031 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,031 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:56:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,034 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4fd9d2[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,036 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,036 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,039 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,039 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,040 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,041 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,045 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,045 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,045 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,046 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,046 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,047 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,047 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,049 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677048049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677048050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677048053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,058 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,058 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,059 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,059 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,059 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 586 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677048139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677048154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677048155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,159 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677048157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,162 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,164 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,164 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,165 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,166 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,166 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,166 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,166 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,167 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a76bc60[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,169 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,170 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,171 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,172 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,174 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,174 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,174 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,177 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,177 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,178 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,181 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:28,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,181 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,182 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,182 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,282 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,285 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,285 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,286 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,287 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,287 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,289 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,289 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,289 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,289 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,289 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,290 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c901400[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,292 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,294 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,295 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,295 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,296 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,299 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,299 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,301 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,301 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,302 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,314 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,314 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,315 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,315 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,315 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,315 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677048360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677048360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677048362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,415 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,418 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,418 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,419 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,420 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,420 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,424 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10924b08[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,426 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,427 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,427 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,429 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=751, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/ec377557fbb046bb9782c128957f46ae 2023-03-24 16:56:28,429 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,434 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,435 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,435 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,437 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,438 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/9ec10e36d5324f29940ad48b20205c3a 2023-03-24 16:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,442 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,445 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,446 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,446 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,446 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,446 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,546 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,550 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,550 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,551 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,552 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,552 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,557 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a19181[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,559 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,560 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,561 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,562 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,565 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,565 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,568 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,568 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,569 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,572 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,572 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:28,573 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,573 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,639 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/1fd399441eba4c84b04ffd699acc1065 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1fd399441eba4c84b04ffd699acc1065 2023-03-24 16:56:28,645 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into 1fd399441eba4c84b04ffd699acc1065(size=26.5 K), total size for store is 52.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:28,645 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:28,645 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=11, startTime=1679676987746; duration=0sec 2023-03-24 16:56:28,645 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677048664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677048666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:28,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677048666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,673 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,676 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,676 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,677 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,678 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,678 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,683 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,685 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ddbbc09[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,686 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,687 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,691 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,694 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,694 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,694 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,695 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,696 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,697 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,699 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,699 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,700 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:28,700 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,800 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,803 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,803 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,804 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,804 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,813 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,814 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,814 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:28,816 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:28,817 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b781a6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:28,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:28,820 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,821 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:28,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,821 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:28,822 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:28,825 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:28,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:28,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:28,825 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:28,825 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:28,827 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,827 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:28,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,827 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,828 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,829 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:28,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:28,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,831 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:28,832 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,831 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:28,831 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:28,832 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:28,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/ea02dd165c434568a39ab8bb561fc945 2023-03-24 16:56:28,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/ec377557fbb046bb9782c128957f46ae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae 2023-03-24 16:56:28,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae, entries=350, sequenceid=751, filesize=52.9 K 2023-03-24 16:56:28,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/9ec10e36d5324f29940ad48b20205c3a as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a 2023-03-24 16:56:28,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a, entries=150, sequenceid=751, filesize=11.9 K 2023-03-24 16:56:28,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/ea02dd165c434568a39ab8bb561fc945 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945 2023-03-24 16:56:28,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945, entries=150, sequenceid=751, filesize=11.9 K 2023-03-24 16:56:28,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa8f591d6f09d5e6894f62e442177704 in 866ms, sequenceid=751, compaction requested=true 2023-03-24 16:56:28,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:28,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:28,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:28,865 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:28,865 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:28,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 16:56:28,866 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107430 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:28,866 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:28,866 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:28,866 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:28,866 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,866 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,867 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1fd399441eba4c84b04ffd699acc1065, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=104.9 K 2023-03-24 16:56:28,867 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f5c7e8b746494c7e8b6071ccd65a1f9f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.8 K 2023-03-24 16:56:28,867 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 1fd399441eba4c84b04ffd699acc1065, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:28,867 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting f5c7e8b746494c7e8b6071ccd65a1f9f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:28,867 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 050940459e1d42699c7ab71d5bbd28c9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1679676986999 2023-03-24 16:56:28,868 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting d44a0333ac1a48fe8401b4bebdeb09e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1679676986999 2023-03-24 16:56:28,868 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting ec377557fbb046bb9782c128957f46ae, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987978 2023-03-24 16:56:28,868 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 9ec10e36d5324f29940ad48b20205c3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987991 2023-03-24 16:56:28,875 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#656 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:28,878 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#655 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:28,897 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/1816c40fd89c4b2499015c8ca8c73efc as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1816c40fd89c4b2499015c8ca8c73efc 2023-03-24 16:56:28,902 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 1816c40fd89c4b2499015c8ca8c73efc(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:28,902 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:28,902 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676988865; duration=0sec 2023-03-24 16:56:28,903 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:28,903 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:28,904 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:28,905 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:28,905 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,905 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ace17bbbe83e48d7b63a738663153e1d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.8 K 2023-03-24 16:56:28,905 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ace17bbbe83e48d7b63a738663153e1d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1679676985750 2023-03-24 16:56:28,905 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 804cb47b1469451ab5314572ef1eaae9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1679676986999 2023-03-24 16:56:28,906 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting ea02dd165c434568a39ab8bb561fc945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987991 2023-03-24 16:56:28,912 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#657 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:28,932 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/e4bb5063ed6a4176a804a3a16b0887f5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e4bb5063ed6a4176a804a3a16b0887f5 2023-03-24 16:56:28,932 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:28,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:28,935 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:28,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:28,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:28,936 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into e4bb5063ed6a4176a804a3a16b0887f5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:28,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,936 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:28,936 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,936 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676988865; duration=0sec 2023-03-24 16:56:28,937 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:28,937 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,937 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:28,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:28,938 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:28,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:28,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:28,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:28,939 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:28,939 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:28,939 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:28,940 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:28,951 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241b81370a605743ad995fa26af193a8e8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241b81370a605743ad995fa26af193a8e8 2023-03-24 16:56:28,957 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=789, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/04e51f9cf20c4158866b417b10a08a49 2023-03-24 16:56:28,970 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/a2191b24b34a4bd6a29de8ab5a68a34c 2023-03-24 16:56:28,982 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/163fc3ab7e8b4f699bae7279031761c0 2023-03-24 16:56:28,985 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/04e51f9cf20c4158866b417b10a08a49 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49 2023-03-24 16:56:28,989 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49, entries=150, sequenceid=789, filesize=25.4 K 2023-03-24 16:56:28,989 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/a2191b24b34a4bd6a29de8ab5a68a34c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c 2023-03-24 16:56:28,992 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c, entries=150, sequenceid=789, filesize=11.9 K 2023-03-24 16:56:28,993 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/163fc3ab7e8b4f699bae7279031761c0 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0 2023-03-24 16:56:28,996 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0, entries=150, sequenceid=789, filesize=11.9 K 2023-03-24 16:56:28,997 INFO [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 57ms, sequenceid=789, compaction requested=false 2023-03-24 16:56:28,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:28,997 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:28,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:28,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:28,997 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,000 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,001 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,002 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,002 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137f5293[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,003 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,005 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,006 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,007 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,010 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,010 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,010 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,011 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,012 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,013 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,015 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,015 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,016 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,016 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,016 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,116 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:29,120 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:29,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:29,120 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:29,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,121 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,123 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,123 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,124 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:29,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:29,125 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,125 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:29,125 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:29,125 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,127 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7cf4e4[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,129 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,130 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,130 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,132 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,134 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,135 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,135 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,138 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,138 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,140 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,143 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,143 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,143 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,143 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:29,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:29,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:29,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677049198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 596 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677049198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677049199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677049200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,244 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:29,247 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:29,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:29,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:29,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,247 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,250 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,251 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:29,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:29,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:29,252 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,252 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:29,252 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:29,252 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,267 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d572cb6[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,269 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,270 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,270 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,270 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,272 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,272 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,274 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,274 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,274 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,276 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,276 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,276 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,276 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,278 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,281 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,281 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,282 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,282 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677049301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 598 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677049302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677049302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677049305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,333 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/812682f7ef8a4e1f8f9ee8494630c7d1 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/812682f7ef8a4e1f8f9ee8494630c7d1 2023-03-24 16:56:29,337 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into 812682f7ef8a4e1f8f9ee8494630c7d1(size=26.6 K), total size for store is 52.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:29,337 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:29,337 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676988865; duration=0sec 2023-03-24 16:56:29,337 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:29,382 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:29,385 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:29,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:29,385 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:29,386 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,386 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,387 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,387 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,388 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:29,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:29,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:29,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:29,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:29,389 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,390 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,390 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,391 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c0320b[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,392 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,393 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,394 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,394 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,395 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,398 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,398 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,398 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,400 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,400 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,400 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,400 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,402 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,404 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,404 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,404 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,404 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,505 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677049504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 600 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677049504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:29,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33315] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677049506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:29,508 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:29,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:29,508 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:29,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33315] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677049507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,509 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,510 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,510 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,511 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:29,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:29,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:29,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:29,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:29,512 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,515 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6942034a[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,517 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,518 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,518 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,519 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,520 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,522 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,522 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,523 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,523 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,524 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,525 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,527 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,530 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,531 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,531 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,530 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,531 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,594 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324bd06cb886cb44608b90e37f45bf97fad to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bd06cb886cb44608b90e37f45bf97fad 2023-03-24 16:56:29,623 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=803, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c9abae524e5a49fca641ba52dcd429f4 2023-03-24 16:56:29,631 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/122ff4973f6e4726b721490b3ba5a476 2023-03-24 16:56:29,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 16:56:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 16:56:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 16:56:29,636 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 16:56:29,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,636 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 16:56:29,637 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 16:56:29,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,640 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,641 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,642 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,642 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,642 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 16:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 16:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 16:56:29,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 16:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 16:56:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 16:56:29,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. started... 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 16:56:29,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. as already flushing 2023-03-24 16:56:29,644 DEBUG [rs(jenkins-hbase4.apache.org,33315,1679676819242)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 16:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 16:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,646 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 16:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33315,1679676819242' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 16:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c31152[Count = 0] remaining members to acquire global barrier 2023-03-24 16:56:29,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 16:56:29,648 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,649 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 16:56:29,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 16:56:29,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,650 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 16:56:29,650 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33315,1679676819242' in zk 2023-03-24 16:56:29,654 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 16:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,33315,1679676819242' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 16:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33315,1679676819242': 2023-03-24 16:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33315,1679676819242' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 16:56:29,658 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 16:56:29,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 16:56:29,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 16:56:29,658 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 16:56:29,658 INFO [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 16:56:29,660 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,660 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 16:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 16:56:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,660 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 16:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,661 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 16:56:29,662 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 16:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 16:56:29,665 DEBUG [(jenkins-hbase4.apache.org,39701,1679676818001)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 16:56:29,664 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 16:56:29,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 16:56:29,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 16:56:29,665 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 16:56:29,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 16:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 16:56:29,665 DEBUG [Thread-4135] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 16:56:29,665 DEBUG [Thread-4135] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 16:56:29,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 16:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 16:56:29,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 16:56:29,697 DEBUG [Thread-4144] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:62195 2023-03-24 16:56:29,697 DEBUG [Thread-4144] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:29,698 DEBUG [Thread-4142] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:62195 2023-03-24 16:56:29,698 DEBUG [Thread-4142] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:29,699 DEBUG [Thread-4140] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:62195 2023-03-24 16:56:29,699 DEBUG [Thread-4140] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:29,699 DEBUG [Thread-4136] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:62195 2023-03-24 16:56:29,699 DEBUG [Thread-4138] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:62195 2023-03-24 16:56:29,700 DEBUG [Thread-4136] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:29,700 DEBUG [Thread-4138] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:29,765 DEBUG [Thread-4135] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 16:56:29,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 16:56:29,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46898 deadline: 1679677049806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 602 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46868 deadline: 1679677049807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46884 deadline: 1679677049811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:29,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 16:56:29,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46912 deadline: 1679677049811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:30,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/356e12229c304b378b3e375953af0fad 2023-03-24 16:56:30,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c9abae524e5a49fca641ba52dcd429f4 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4 2023-03-24 16:56:30,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4, entries=250, sequenceid=803, filesize=39.2 K 2023-03-24 16:56:30,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/122ff4973f6e4726b721490b3ba5a476 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476 2023-03-24 16:56:30,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476, entries=150, sequenceid=803, filesize=11.9 K 2023-03-24 16:56:30,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/356e12229c304b378b3e375953af0fad as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad 2023-03-24 16:56:30,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad, entries=150, sequenceid=803, filesize=11.9 K 2023-03-24 16:56:30,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fa8f591d6f09d5e6894f62e442177704 in 874ms, sequenceid=803, compaction requested=true 2023-03-24 16:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:30,055 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:30,055 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:30,056 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93448 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:30,056 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:30,056 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/A is initiating minor compaction (all files) 2023-03-24 16:56:30,056 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/B is initiating minor compaction (all files) 2023-03-24 16:56:30,056 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/A in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:30,056 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/B in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:30,056 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/812682f7ef8a4e1f8f9ee8494630c7d1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=91.3 K 2023-03-24 16:56:30,056 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1816c40fd89c4b2499015c8ca8c73efc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.9 K 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 812682f7ef8a4e1f8f9ee8494630c7d1, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987991 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 1816c40fd89c4b2499015c8ca8c73efc, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987991 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting 04e51f9cf20c4158866b417b10a08a49, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1679676988048 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting a2191b24b34a4bd6a29de8ab5a68a34c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1679676988048 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] compactions.Compactor(207): Compacting c9abae524e5a49fca641ba52dcd429f4, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=803, earliestPutTs=1679676989176 2023-03-24 16:56:30,057 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 122ff4973f6e4726b721490b3ba5a476, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=803, earliestPutTs=1679676989180 2023-03-24 16:56:30,062 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#B#compaction#664 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:30,065 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#A#compaction#665 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33315] regionserver.HRegion(9158): Flush requested on fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:30,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 16:56:30,311 DEBUG [Thread-4133] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:62195 2023-03-24 16:56:30,311 DEBUG [Thread-4133] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:30,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:30,312 DEBUG [Thread-4125] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:62195 2023-03-24 16:56:30,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:30,312 DEBUG [Thread-4125] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:30,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:30,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:30,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:30,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 16:56:30,317 DEBUG [Thread-4131] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:62195 2023-03-24 16:56:30,317 DEBUG [Thread-4131] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:30,318 DEBUG [Thread-4127] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:62195 2023-03-24 16:56:30,318 DEBUG [Thread-4127] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:30,469 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/27d098eb2433452fb477b5470783c15b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/27d098eb2433452fb477b5470783c15b 2023-03-24 16:56:30,473 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/B of fa8f591d6f09d5e6894f62e442177704 into 27d098eb2433452fb477b5470783c15b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:30,473 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:30,473 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/B, priority=13, startTime=1679676990055; duration=0sec 2023-03-24 16:56:30,473 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 16:56:30,473 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 16:56:30,474 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 16:56:30,474 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1912): fa8f591d6f09d5e6894f62e442177704/C is initiating minor compaction (all files) 2023-03-24 16:56:30,474 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa8f591d6f09d5e6894f62e442177704/C in TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:30,474 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e4bb5063ed6a4176a804a3a16b0887f5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad] into tmpdir=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp, totalSize=36.9 K 2023-03-24 16:56:30,474 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting e4bb5063ed6a4176a804a3a16b0887f5, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679676987991 2023-03-24 16:56:30,475 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 163fc3ab7e8b4f699bae7279031761c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1679676988048 2023-03-24 16:56:30,475 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] compactions.Compactor(207): Compacting 356e12229c304b378b3e375953af0fad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=803, earliestPutTs=1679676989180 2023-03-24 16:56:30,479 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] throttle.PressureAwareThroughputController(145): fa8f591d6f09d5e6894f62e442177704#C#compaction#667 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 16:56:30,723 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c0fefd2a014a4470b8337cfc1bb7c584 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c0fefd2a014a4470b8337cfc1bb7c584 2023-03-24 16:56:30,888 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/770f1cfa47144fc38034e3038a5ad1a5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/770f1cfa47144fc38034e3038a5ad1a5 2023-03-24 16:56:30,892 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/C of fa8f591d6f09d5e6894f62e442177704 into 770f1cfa47144fc38034e3038a5ad1a5(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 16:56:30,892 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:30,892 INFO [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/C, priority=13, startTime=1679676990055; duration=0sec 2023-03-24 16:56:30,892 DEBUG [RS:0;jenkins-hbase4:33315-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:31,128 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=825, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/31b6da0588014fcfa7e58731e17a122c 2023-03-24 16:56:31,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b55798ceb159444487a1698a69df1ff5 2023-03-24 16:56:31,280 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/c6716f37908f480fa51d30dd01035b87 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c6716f37908f480fa51d30dd01035b87 2023-03-24 16:56:31,283 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fa8f591d6f09d5e6894f62e442177704/A of fa8f591d6f09d5e6894f62e442177704 into c6716f37908f480fa51d30dd01035b87(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 16:56:31,283 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:31,283 INFO [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704., storeName=fa8f591d6f09d5e6894f62e442177704/A, priority=13, startTime=1679676990055; duration=1sec 2023-03-24 16:56:31,284 DEBUG [RS:0;jenkins-hbase4:33315-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 16:56:31,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b67facf58947435a8f6f52669b96bc26 2023-03-24 16:56:31,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/31b6da0588014fcfa7e58731e17a122c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/31b6da0588014fcfa7e58731e17a122c 2023-03-24 16:56:31,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/31b6da0588014fcfa7e58731e17a122c, entries=150, sequenceid=825, filesize=25.4 K 2023-03-24 16:56:31,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/b55798ceb159444487a1698a69df1ff5 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b55798ceb159444487a1698a69df1ff5 2023-03-24 16:56:31,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b55798ceb159444487a1698a69df1ff5, entries=150, sequenceid=825, filesize=11.9 K 2023-03-24 16:56:31,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/b67facf58947435a8f6f52669b96bc26 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b67facf58947435a8f6f52669b96bc26 2023-03-24 16:56:31,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b67facf58947435a8f6f52669b96bc26, entries=150, sequenceid=825, filesize=11.9 K 2023-03-24 16:56:31,558 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=13.42 KB/13740 for fa8f591d6f09d5e6894f62e442177704 in 1247ms, sequenceid=825, compaction requested=false 2023-03-24 16:56:31,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:37,297 DEBUG [Thread-4129] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:62195 2023-03-24 16:56:37,297 DEBUG [Thread-4129] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(394): wrote 469 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4197 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4139 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4157 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4205 2023-03-24 16:56:37,297 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(398): read 4149 2023-03-24 16:56:37,298 INFO [Listener at localhost/43391] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 16:56:37,298 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:56:37,298 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:62195 2023-03-24 16:56:37,298 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:37,298 INFO [Listener at localhost/43391] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 16:56:37,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 16:56:37,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:37,301 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676997301"}]},"ts":"1679676997301"} 2023-03-24 16:56:37,302 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 16:56:37,304 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 16:56:37,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, UNASSIGN}] 2023-03-24 16:56:37,306 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, UNASSIGN 2023-03-24 16:56:37,306 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:37,307 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676997306"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679676997306"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679676997306"}]},"ts":"1679676997306"} 2023-03-24 16:56:37,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242}] 2023-03-24 16:56:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:37,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:37,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fa8f591d6f09d5e6894f62e442177704, disabling compactions & flushes 2023-03-24 16:56:37,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:37,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. after waiting 0 ms 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:37,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fa8f591d6f09d5e6894f62e442177704 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=A 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=B 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa8f591d6f09d5e6894f62e442177704, store=C 2023-03-24 16:56:37,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 16:56:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:37,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d576be6e58b8499ab876c11b0ddb7ebd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d576be6e58b8499ab876c11b0ddb7ebd 2023-03-24 16:56:37,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:38,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=834, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9b41ba8649e641248d68fd5fac18fd8b 2023-03-24 16:56:38,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/a08cc315f85d41e29d080e2646afc603 2023-03-24 16:56:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:38,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/72b21d3abe634d7e8d333519760646ae 2023-03-24 16:56:38,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/A/9b41ba8649e641248d68fd5fac18fd8b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9b41ba8649e641248d68fd5fac18fd8b 2023-03-24 16:56:38,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9b41ba8649e641248d68fd5fac18fd8b, entries=100, sequenceid=834, filesize=18.6 K 2023-03-24 16:56:38,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/B/a08cc315f85d41e29d080e2646afc603 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a08cc315f85d41e29d080e2646afc603 2023-03-24 16:56:38,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a08cc315f85d41e29d080e2646afc603, entries=100, sequenceid=834, filesize=9.5 K 2023-03-24 16:56:38,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/.tmp/C/72b21d3abe634d7e8d333519760646ae as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/72b21d3abe634d7e8d333519760646ae 2023-03-24 16:56:38,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/72b21d3abe634d7e8d333519760646ae, entries=100, sequenceid=834, filesize=9.5 K 2023-03-24 16:56:38,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for fa8f591d6f09d5e6894f62e442177704 in 1242ms, sequenceid=834, compaction requested=true 2023-03-24 16:56:38,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d9150d015dbf423da722850290c6366e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/425960ee6ba74ccd817d8b812621a616, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/cd4c86c6191343cfa95f6072585934c6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a86cd49609b945818263b532d38b5048, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c3d66e47dd2241d099cb6d7472e32d7e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4bad458673964441b66512d70b3a41b9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e4665b2c8d3443e5bd5729e3bc794c63, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5c6e2e93d7c449acb18927f884bfb309, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1fd399441eba4c84b04ffd699acc1065, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/812682f7ef8a4e1f8f9ee8494630c7d1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4] to archive 2023-03-24 16:56:38,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:38,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/017741500ebb43f7b5609528a87d618e 2023-03-24 16:56:38,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9e8b05dd98cd42b9ac6d5a51161d937f 2023-03-24 16:56:38,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d9150d015dbf423da722850290c6366e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d9150d015dbf423da722850290c6366e 2023-03-24 16:56:38,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/d5b490f4e24c4c82bb9a4c02ad665bfe 2023-03-24 16:56:38,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a87f4571680c44cea0b3edfebc40b665 2023-03-24 16:56:38,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/425960ee6ba74ccd817d8b812621a616 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/425960ee6ba74ccd817d8b812621a616 2023-03-24 16:56:38,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/caa2b840c1534410b3e65d505cc6f2f8 2023-03-24 16:56:38,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/77a2ad3f34454f92adf78630e63dbf94 2023-03-24 16:56:38,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9a9da088804b4c279f9459f3f21e0df3 2023-03-24 16:56:38,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/cd4c86c6191343cfa95f6072585934c6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/cd4c86c6191343cfa95f6072585934c6 2023-03-24 16:56:38,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c44c58d762ec4eec8e040cba7dbf0efc 2023-03-24 16:56:38,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b2427f1e0f754c01af84689a281a8d88 2023-03-24 16:56:38,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a86cd49609b945818263b532d38b5048 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/a86cd49609b945818263b532d38b5048 2023-03-24 16:56:38,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4e2919d209614ccd8ef9b67420243309 2023-03-24 16:56:38,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7760604c8f4b41f39543a10c0381495a 2023-03-24 16:56:38,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/dfff822ccedf4ee3bc43eefe6adad152 2023-03-24 16:56:38,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c3d66e47dd2241d099cb6d7472e32d7e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c3d66e47dd2241d099cb6d7472e32d7e 2023-03-24 16:56:38,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e21d5bdba75d4b0dad4bee907b404e20 2023-03-24 16:56:38,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1ebdad2ba8cc4f6b8e69ad3132edc587 2023-03-24 16:56:38,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4bad458673964441b66512d70b3a41b9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/4bad458673964441b66512d70b3a41b9 2023-03-24 16:56:38,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/8dfdac030e6c49278063355d22b5675c 2023-03-24 16:56:38,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5654510b990e4c2ca0640d4d25dc3cfa 2023-03-24 16:56:38,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/45d78a197eed4909b5cc7b8d9ef00382 2023-03-24 16:56:38,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e4665b2c8d3443e5bd5729e3bc794c63 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/e4665b2c8d3443e5bd5729e3bc794c63 2023-03-24 16:56:38,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7e64d3e3230f491f9230775fe1cab63f 2023-03-24 16:56:38,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/b496c06bee144236a7de10a4e0183b28 2023-03-24 16:56:38,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/f7f005b006464fe697622d2e6711ce93 2023-03-24 16:56:38,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b1b9f308df842ee9e189ffada3c55c6 2023-03-24 16:56:38,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5c6e2e93d7c449acb18927f884bfb309 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/5c6e2e93d7c449acb18927f884bfb309 2023-03-24 16:56:38,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/3b8c0a753e044459b59c2ed0246fded5 2023-03-24 16:56:38,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/7a5d3397647d4d1f903895f08c20171c 2023-03-24 16:56:38,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/542f5b45aabc41b2b7a99b7f503f4877 2023-03-24 16:56:38,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/43edaec48d9648ccb15cbfd8dfbbafb2 2023-03-24 16:56:38,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1fd399441eba4c84b04ffd699acc1065 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/1fd399441eba4c84b04ffd699acc1065 2023-03-24 16:56:38,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/050940459e1d42699c7ab71d5bbd28c9 2023-03-24 16:56:38,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/ec377557fbb046bb9782c128957f46ae 2023-03-24 16:56:38,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/812682f7ef8a4e1f8f9ee8494630c7d1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/812682f7ef8a4e1f8f9ee8494630c7d1 2023-03-24 16:56:38,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/04e51f9cf20c4158866b417b10a08a49 2023-03-24 16:56:38,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c9abae524e5a49fca641ba52dcd429f4 2023-03-24 16:56:38,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ae3f76cee2e848478989a26497e53c86, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/007b1a97b06243249b1ec4ce671cfdd7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f323698c620e4d318683f425fe5f32bb, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1c799e5ef20d40578c44efeb51831bda, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2ea50b3d81e741cd8255480b03a73ddd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3b5ba07803a046c2b9937f2338c2ff62, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/7a1d193c77c546c9b72cce32b6e7fca3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fa1d540486144d4285cd0e0a397415ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f5c7e8b746494c7e8b6071ccd65a1f9f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1816c40fd89c4b2499015c8ca8c73efc, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476] to archive 2023-03-24 16:56:38,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:38,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/8caf44ac28f54fd3b16d7fc83b99f2cd 2023-03-24 16:56:38,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1b184e82317f4f2d9f5a3ac93db9f7c3 2023-03-24 16:56:38,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ae3f76cee2e848478989a26497e53c86 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ae3f76cee2e848478989a26497e53c86 2023-03-24 16:56:38,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/5de278e14f6d4ecdae321f4b88306122 2023-03-24 16:56:38,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/13b887cab1b148a9965bd554005e6d9e 2023-03-24 16:56:38,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/007b1a97b06243249b1ec4ce671cfdd7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/007b1a97b06243249b1ec4ce671cfdd7 2023-03-24 16:56:38,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/e1fd48f99f4045a39d0f8f2c5353a7ca 2023-03-24 16:56:38,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/be73d0a568964a2b847c2bbdc5f48d0d 2023-03-24 16:56:38,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f323698c620e4d318683f425fe5f32bb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f323698c620e4d318683f425fe5f32bb 2023-03-24 16:56:38,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/6b8e503dc53b4fcd9d7b8da8152ee63f 2023-03-24 16:56:38,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3d8b1d4b503e43f2a7d1973f31b611ba 2023-03-24 16:56:38,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1c799e5ef20d40578c44efeb51831bda to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1c799e5ef20d40578c44efeb51831bda 2023-03-24 16:56:38,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/4df62e41c368426ea95462bb336aa362 2023-03-24 16:56:38,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d427831229854c77ae34a10e9b4146f8 2023-03-24 16:56:38,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2693a6fcb73c49b3978eac9b81340fa0 2023-03-24 16:56:38,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/c22cff962528453785cff726abae81c6 2023-03-24 16:56:38,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2ea50b3d81e741cd8255480b03a73ddd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/2ea50b3d81e741cd8255480b03a73ddd 2023-03-24 16:56:38,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f525653dd3df4403aaccfb6bd84f831e 2023-03-24 16:56:38,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/26a3e367180c474091df2652bf831745 2023-03-24 16:56:38,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3b5ba07803a046c2b9937f2338c2ff62 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3b5ba07803a046c2b9937f2338c2ff62 2023-03-24 16:56:38,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3c97f1d151264b3d86f50805d1f95646 2023-03-24 16:56:38,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ff48f5bbf4ae4ee1a9decfc3934aaa13 2023-03-24 16:56:38,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/7a1d193c77c546c9b72cce32b6e7fca3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/7a1d193c77c546c9b72cce32b6e7fca3 2023-03-24 16:56:38,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d71dde924b514395a1247e0dd4967e83 2023-03-24 16:56:38,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/48d5562dc36f4118875adafac7c43d4e 2023-03-24 16:56:38,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/ba4fa1a837d64d0a92c695c8f7a83df7 2023-03-24 16:56:38,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/74556d2dba24412389a2039b91155faf 2023-03-24 16:56:38,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fa1d540486144d4285cd0e0a397415ae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fa1d540486144d4285cd0e0a397415ae 2023-03-24 16:56:38,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/3413521963594b9c94169f31c75dbca2 2023-03-24 16:56:38,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/fc00f885f9e9491fbff236aeed066515 2023-03-24 16:56:38,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b64518d757ee44d5bd3580af5cf67ea2 2023-03-24 16:56:38,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/72fbc3de227a4dd196d762aca4957a82 2023-03-24 16:56:38,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f5c7e8b746494c7e8b6071ccd65a1f9f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/f5c7e8b746494c7e8b6071ccd65a1f9f 2023-03-24 16:56:38,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b20c7118409943e3b4f139cf9d6a47f2 2023-03-24 16:56:38,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/d44a0333ac1a48fe8401b4bebdeb09e9 2023-03-24 16:56:38,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1816c40fd89c4b2499015c8ca8c73efc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/1816c40fd89c4b2499015c8ca8c73efc 2023-03-24 16:56:38,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/9ec10e36d5324f29940ad48b20205c3a 2023-03-24 16:56:38,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a2191b24b34a4bd6a29de8ab5a68a34c 2023-03-24 16:56:38,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/122ff4973f6e4726b721490b3ba5a476 2023-03-24 16:56:38,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fbf05197ae634907952869d5ac45ae09, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/879d48c0ce564bafbd69ba68afee3b59, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4e50e3ebcce74ce9a94aab1dca26d325, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4c56df83945649959551fd82ea5ae3ae, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cf89e6b9cf824afd9056ca4f3b3d97ca, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5d831db8097f4269b7bf0a76b4570d52, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/04ecb305fd6d40eebed7fbeab0d668df, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ace17bbbe83e48d7b63a738663153e1d, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e4bb5063ed6a4176a804a3a16b0887f5, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad] to archive 2023-03-24 16:56:38,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 16:56:38,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c4ec87346947d89e461555e6a5f0e6 2023-03-24 16:56:38,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/34ec80c1a07e4ba984201bd79e009914 2023-03-24 16:56:38,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fbf05197ae634907952869d5ac45ae09 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fbf05197ae634907952869d5ac45ae09 2023-03-24 16:56:38,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4d7ccc3a16f44be49ebc1eb97d0e1af2 2023-03-24 16:56:38,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b3c3746f649d4b7182e1e70c5f558eae 2023-03-24 16:56:38,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/879d48c0ce564bafbd69ba68afee3b59 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/879d48c0ce564bafbd69ba68afee3b59 2023-03-24 16:56:38,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ce1a95a9717f41a9b566fcf23db44ae3 2023-03-24 16:56:38,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/83d1fbfe6a7844ea9fa7d08c1461f8b4 2023-03-24 16:56:38,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e9ff342e82684b77b3faa274219d9a06 2023-03-24 16:56:38,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4e50e3ebcce74ce9a94aab1dca26d325 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4e50e3ebcce74ce9a94aab1dca26d325 2023-03-24 16:56:38,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/442539240bf145d087b61645589f13f2 2023-03-24 16:56:38,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/366ae493198f4a489be0346ea2a73099 2023-03-24 16:56:38,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/fdd9ba6c76f24746ae3bcb9a14c1e4b5 2023-03-24 16:56:38,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/6b7e7fc3e2ad4eab8784e49749dc87a1 2023-03-24 16:56:38,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/1d624438899a45df99ffb55e10b4e693 2023-03-24 16:56:38,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4c56df83945649959551fd82ea5ae3ae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/4c56df83945649959551fd82ea5ae3ae 2023-03-24 16:56:38,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/f757e0f462744cffa680b23217cbf933 2023-03-24 16:56:38,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/337ec43eb9cd4edb84fec7b16085fa62 2023-03-24 16:56:38,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cf89e6b9cf824afd9056ca4f3b3d97ca to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cf89e6b9cf824afd9056ca4f3b3d97ca 2023-03-24 16:56:38,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a8c6c35dc37b4267b984a5e8056cf06f 2023-03-24 16:56:38,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a380eaedf87e473ea67fe5f279da7807 2023-03-24 16:56:38,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5d831db8097f4269b7bf0a76b4570d52 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5d831db8097f4269b7bf0a76b4570d52 2023-03-24 16:56:38,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/5f16f299e7b840819ed25c0062a6f532 2023-03-24 16:56:38,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/59819dedd001415cacdac633bf44c86f 2023-03-24 16:56:38,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/79db461905644002b3825eb59f623590 2023-03-24 16:56:38,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e92246120b7c4ded8bd68fd195fdfa31 2023-03-24 16:56:38,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/cc645a3a944b45ec89ef8d88a5e18d26 2023-03-24 16:56:38,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/04ecb305fd6d40eebed7fbeab0d668df to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/04ecb305fd6d40eebed7fbeab0d668df 2023-03-24 16:56:38,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/a03b2132c8544654b70ac72a3202d06c 2023-03-24 16:56:38,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b18e909124dc402bb81eda80aca0b4dd 2023-03-24 16:56:38,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/39118033f6d045f18dbdf54872630c75 2023-03-24 16:56:38,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ace17bbbe83e48d7b63a738663153e1d to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ace17bbbe83e48d7b63a738663153e1d 2023-03-24 16:56:38,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/89e72baa66274f58ba62e995f46a9b50 2023-03-24 16:56:38,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/804cb47b1469451ab5314572ef1eaae9 2023-03-24 16:56:38,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e4bb5063ed6a4176a804a3a16b0887f5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/e4bb5063ed6a4176a804a3a16b0887f5 2023-03-24 16:56:38,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/ea02dd165c434568a39ab8bb561fc945 2023-03-24 16:56:38,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/163fc3ab7e8b4f699bae7279031761c0 2023-03-24 16:56:38,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/356e12229c304b378b3e375953af0fad 2023-03-24 16:56:38,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits/837.seqid, newMaxSeqId=837, maxSeqId=4 2023-03-24 16:56:38,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704. 2023-03-24 16:56:38,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fa8f591d6f09d5e6894f62e442177704: 2023-03-24 16:56:38,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:38,839 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=fa8f591d6f09d5e6894f62e442177704, regionState=CLOSED 2023-03-24 16:56:38,839 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679676998839"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679676998839"}]},"ts":"1679676998839"} 2023-03-24 16:56:38,841 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-24 16:56:38,841 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure fa8f591d6f09d5e6894f62e442177704, server=jenkins-hbase4.apache.org,33315,1679676819242 in 1.5320 sec 2023-03-24 16:56:38,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-24 16:56:38,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa8f591d6f09d5e6894f62e442177704, UNASSIGN in 1.5360 sec 2023-03-24 16:56:38,843 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679676998843"}]},"ts":"1679676998843"} 2023-03-24 16:56:38,843 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 16:56:38,845 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 16:56:38,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5460 sec 2023-03-24 16:56:39,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 16:56:39,406 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-24 16:56:39,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 16:56:39,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,408 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,409 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 16:56:39,412 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:39,414 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C, FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits] 2023-03-24 16:56:39,416 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/31b6da0588014fcfa7e58731e17a122c to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/31b6da0588014fcfa7e58731e17a122c 2023-03-24 16:56:39,417 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9b41ba8649e641248d68fd5fac18fd8b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/9b41ba8649e641248d68fd5fac18fd8b 2023-03-24 16:56:39,418 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c6716f37908f480fa51d30dd01035b87 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/A/c6716f37908f480fa51d30dd01035b87 2023-03-24 16:56:39,420 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/27d098eb2433452fb477b5470783c15b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/27d098eb2433452fb477b5470783c15b 2023-03-24 16:56:39,421 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a08cc315f85d41e29d080e2646afc603 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/a08cc315f85d41e29d080e2646afc603 2023-03-24 16:56:39,422 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b55798ceb159444487a1698a69df1ff5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/B/b55798ceb159444487a1698a69df1ff5 2023-03-24 16:56:39,424 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/72b21d3abe634d7e8d333519760646ae to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/72b21d3abe634d7e8d333519760646ae 2023-03-24 16:56:39,425 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/770f1cfa47144fc38034e3038a5ad1a5 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/770f1cfa47144fc38034e3038a5ad1a5 2023-03-24 16:56:39,425 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b67facf58947435a8f6f52669b96bc26 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/C/b67facf58947435a8f6f52669b96bc26 2023-03-24 16:56:39,428 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits/837.seqid to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704/recovered.edits/837.seqid 2023-03-24 16:56:39,428 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/.tmp/data/default/TestAcidGuarantees/fa8f591d6f09d5e6894f62e442177704 2023-03-24 16:56:39,428 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 16:56:39,429 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:56:39,429 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 16:56:39,432 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032401ef2bfa5e734a9c84c2ba5d3bbb204f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032401ef2bfa5e734a9c84c2ba5d3bbb204f 2023-03-24 16:56:39,433 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241b81370a605743ad995fa26af193a8e8 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241b81370a605743ad995fa26af193a8e8 2023-03-24 16:56:39,434 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241dbf374476ca4b2aa9cd0a51e82a4b51 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241dbf374476ca4b2aa9cd0a51e82a4b51 2023-03-24 16:56:39,435 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241fc6841e2e2044ed9ae600ae8e2e5789 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241fc6841e2e2044ed9ae600ae8e2e5789 2023-03-24 16:56:39,436 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422eaab69f6124b70980377ef26a6c832 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032422eaab69f6124b70980377ef26a6c832 2023-03-24 16:56:39,437 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324273c58a71d8e4708b265163720fd922b to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324273c58a71d8e4708b265163720fd922b 2023-03-24 16:56:39,438 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428be81ae79794efaaf3618762caad8eb to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428be81ae79794efaaf3618762caad8eb 2023-03-24 16:56:39,438 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242edaa9d171ee4606ad5e674fa0d5f041 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242edaa9d171ee4606ad5e674fa0d5f041 2023-03-24 16:56:39,439 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243aef2d4176fb407ca21f4430a13d0b75 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243aef2d4176fb407ca21f4430a13d0b75 2023-03-24 16:56:39,440 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245920713ff10a4240af49184e771280b2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245920713ff10a4240af49184e771280b2 2023-03-24 16:56:39,441 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245f402c6a682541eabee4a15c587facc4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245f402c6a682541eabee4a15c587facc4 2023-03-24 16:56:39,442 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246fe0bae4a5c64a7c8d4cd6ad26200ad6 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246fe0bae4a5c64a7c8d4cd6ad26200ad6 2023-03-24 16:56:39,443 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032483617b8424604b35856e54e9e1fbc4e4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032483617b8424604b35856e54e9e1fbc4e4 2023-03-24 16:56:39,443 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248951189c748440af9cf8419b12c05190 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248951189c748440af9cf8419b12c05190 2023-03-24 16:56:39,444 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fc7314e85904642bb9a93b7e8c79858 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248fc7314e85904642bb9a93b7e8c79858 2023-03-24 16:56:39,445 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249171f0f7f6a149deaec0eb9bc3bc6ce3 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249171f0f7f6a149deaec0eb9bc3bc6ce3 2023-03-24 16:56:39,446 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032498ba4a1bc8f84847ad6c9a19bd2835ab to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032498ba4a1bc8f84847ad6c9a19bd2835ab 2023-03-24 16:56:39,446 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f00d0a6fabe41eba1d89ae62d44e489 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f00d0a6fabe41eba1d89ae62d44e489 2023-03-24 16:56:39,447 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a40bf82a850447898d29a6b2cd75203a to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a40bf82a850447898d29a6b2cd75203a 2023-03-24 16:56:39,448 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324aac9855c7fb141adbd0d63d3ac0823d7 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324aac9855c7fb141adbd0d63d3ac0823d7 2023-03-24 16:56:39,449 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac9ea17883d0425a8e1b8eb4b3fd0420 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac9ea17883d0425a8e1b8eb4b3fd0420 2023-03-24 16:56:39,449 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ae51b8caa9bf45f3b0aeb05983c760b4 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ae51b8caa9bf45f3b0aeb05983c760b4 2023-03-24 16:56:39,450 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bd06cb886cb44608b90e37f45bf97fad to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bd06cb886cb44608b90e37f45bf97fad 2023-03-24 16:56:39,451 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324be66b10fcba8453a9410a05574e585f2 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324be66b10fcba8453a9410a05574e585f2 2023-03-24 16:56:39,452 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c0fefd2a014a4470b8337cfc1bb7c584 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c0fefd2a014a4470b8337cfc1bb7c584 2023-03-24 16:56:39,453 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cd88711926de4e6eb4a99d3de7a20f32 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cd88711926de4e6eb4a99d3de7a20f32 2023-03-24 16:56:39,453 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d413cc7a4eff4583bdcc6361495f3150 to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d413cc7a4eff4583bdcc6361495f3150 2023-03-24 16:56:39,454 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d576be6e58b8499ab876c11b0ddb7ebd to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d576be6e58b8499ab876c11b0ddb7ebd 2023-03-24 16:56:39,455 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3be999fac7041c88c06752ff0b61bef to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3be999fac7041c88c06752ff0b61bef 2023-03-24 16:56:39,456 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324edea393225b9475ca1e7a535fc4a9bdc to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324edea393225b9475ca1e7a535fc4a9bdc 2023-03-24 16:56:39,456 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7222389cb5a406487c4e2bf1045e87f to hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7222389cb5a406487c4e2bf1045e87f 2023-03-24 16:56:39,457 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 16:56:39,459 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,460 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 16:56:39,461 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 16:56:39,462 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,462 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 16:56:39,462 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679676999462"}]},"ts":"9223372036854775807"} 2023-03-24 16:56:39,463 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 16:56:39,464 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fa8f591d6f09d5e6894f62e442177704, NAME => 'TestAcidGuarantees,,1679676966606.fa8f591d6f09d5e6894f62e442177704.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 16:56:39,464 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 16:56:39,464 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679676999464"}]},"ts":"9223372036854775807"} 2023-03-24 16:56:39,465 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 16:56:39,466 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 16:56:39,467 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 60 msec 2023-03-24 16:56:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39701] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 16:56:39,510 INFO [Listener at localhost/43391] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-24 16:56:39,524 INFO [Listener at localhost/43391] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 233), OpenFileDescriptor=567 (was 606), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=380 (was 361) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=5762 (was 5689) - AvailableMemoryMB LEAK? - 2023-03-24 16:56:39,524 INFO [Listener at localhost/43391] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-24 16:56:39,524 INFO [Listener at localhost/43391] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 16:56:39,526 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69cada93 to 127.0.0.1:62195 2023-03-24 16:56:39,526 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:39,526 DEBUG [Listener at localhost/43391] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-24 16:56:39,526 DEBUG [Listener at localhost/43391] util.JVMClusterUtil(257): Found active master hash=636902043, stopped=false 2023-03-24 16:56:39,526 INFO [Listener at localhost/43391] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:56:39,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 16:56:39,527 INFO [Listener at localhost/43391] procedure2.ProcedureExecutor(629): Stopping 2023-03-24 16:56:39,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 16:56:39,527 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:56:39,528 DEBUG [Listener at localhost/43391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x660fb749 to 127.0.0.1:62195 2023-03-24 16:56:39,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 16:56:39,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 16:56:39,528 DEBUG [Listener at localhost/43391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:39,528 INFO [Listener at localhost/43391] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33315,1679676819242' ***** 2023-03-24 16:56:39,528 INFO [Listener at localhost/43391] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-24 16:56:39,529 INFO [RS:0;jenkins-hbase4:33315] regionserver.HeapMemoryManager(220): Stopping 2023-03-24 16:56:39,529 INFO [RS:0;jenkins-hbase4:33315] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-24 16:56:39,529 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-24 16:56:39,529 INFO [RS:0;jenkins-hbase4:33315] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-24 16:56:39,529 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(3304): Received CLOSE for 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:39,530 DEBUG [RS:0;jenkins-hbase4:33315] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c3b4380 to 127.0.0.1:62195 2023-03-24 16:56:39,530 DEBUG [RS:0;jenkins-hbase4:33315] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-24 16:56:39,530 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-24 16:56:39,530 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 8b973ca11f7cca1ffdda5919062ee7c1=hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1.} 2023-03-24 16:56:39,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8b973ca11f7cca1ffdda5919062ee7c1, disabling compactions & flushes 2023-03-24 16:56:39,531 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 16:56:39,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:56:39,532 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. after waiting 0 ms 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:56:39,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 16:56:39,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8b973ca11f7cca1ffdda5919062ee7c1 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-24 16:56:39,532 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-24 16:56:39,533 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:56:39,543 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/info/e6d4aaad983a45e8814b5b01758b152c 2023-03-24 16:56:39,547 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e6d4aaad983a45e8814b5b01758b152c 2023-03-24 16:56:39,576 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-24 16:56:39,733 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:56:39,933 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b973ca11f7cca1ffdda5919062ee7c1 2023-03-24 16:56:39,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/.tmp/info/cfc8c6dcf1454e50b8b3c1a4b6d76c9d 2023-03-24 16:56:39,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/.tmp/info/cfc8c6dcf1454e50b8b3c1a4b6d76c9d as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/info/cfc8c6dcf1454e50b8b3c1a4b6d76c9d 2023-03-24 16:56:39,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/info/cfc8c6dcf1454e50b8b3c1a4b6d76c9d, entries=2, sequenceid=6, filesize=4.8 K 2023-03-24 16:56:39,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 8b973ca11f7cca1ffdda5919062ee7c1 in 418ms, sequenceid=6, compaction requested=false 2023-03-24 16:56:39,953 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/rep_barrier/dc6a53dfefb5491a8017430e752f60fa 2023-03-24 16:56:39,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/namespace/8b973ca11f7cca1ffdda5919062ee7c1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-24 16:56:39,955 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dc6a53dfefb5491a8017430e752f60fa 2023-03-24 16:56:39,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:56:39,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8b973ca11f7cca1ffdda5919062ee7c1: 2023-03-24 16:56:39,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679676821527.8b973ca11f7cca1ffdda5919062ee7c1. 2023-03-24 16:56:40,007 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-24 16:56:40,007 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-24 16:56:40,134 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 16:56:40,334 DEBUG [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 16:56:40,361 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/table/501beb2253054582ba505eb24790612b 2023-03-24 16:56:40,365 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 501beb2253054582ba505eb24790612b 2023-03-24 16:56:40,365 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/info/e6d4aaad983a45e8814b5b01758b152c as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/info/e6d4aaad983a45e8814b5b01758b152c 2023-03-24 16:56:40,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e6d4aaad983a45e8814b5b01758b152c 2023-03-24 16:56:40,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/info/e6d4aaad983a45e8814b5b01758b152c, entries=22, sequenceid=93, filesize=7.1 K 2023-03-24 16:56:40,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/rep_barrier/dc6a53dfefb5491a8017430e752f60fa as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/rep_barrier/dc6a53dfefb5491a8017430e752f60fa 2023-03-24 16:56:40,372 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dc6a53dfefb5491a8017430e752f60fa 2023-03-24 16:56:40,373 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/rep_barrier/dc6a53dfefb5491a8017430e752f60fa, entries=6, sequenceid=93, filesize=5.5 K 2023-03-24 16:56:40,373 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/.tmp/table/501beb2253054582ba505eb24790612b as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/table/501beb2253054582ba505eb24790612b 2023-03-24 16:56:40,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 501beb2253054582ba505eb24790612b 2023-03-24 16:56:40,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/table/501beb2253054582ba505eb24790612b, entries=9, sequenceid=93, filesize=5.5 K 2023-03-24 16:56:40,376 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 844ms, sequenceid=93, compaction requested=false 2023-03-24 16:56:40,383 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-24 16:56:40,383 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 16:56:40,384 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 16:56:40,384 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 16:56:40,384 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-24 16:56:40,534 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33315,1679676819242; all regions closed. 2023-03-24 16:56:40,542 DEBUG [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/oldWALs 2023-03-24 16:56:40,542 INFO [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33315%2C1679676819242.meta:.meta(num 1679676821262) 2023-03-24 16:56:40,546 DEBUG [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/oldWALs 2023-03-24 16:56:40,546 INFO [RS:0;jenkins-hbase4:33315] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33315%2C1679676819242:(num 1679676821087) 2023-03-24 16:56:40,546 DEBUG [RS:0;jenkins-hbase4:33315] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:40,546 INFO [RS:0;jenkins-hbase4:33315] regionserver.LeaseManager(133): Closed leases 2023-03-24 16:56:40,546 INFO [RS:0;jenkins-hbase4:33315] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-24 16:56:40,546 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 16:56:40,548 INFO [RS:0;jenkins-hbase4:33315] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33315 2023-03-24 16:56:40,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33315,1679676819242 2023-03-24 16:56:40,576 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 16:56:40,577 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@791bc874 rejected from java.util.concurrent.ThreadPoolExecutor@7d24538f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2773] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,577 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 16:56:40,578 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2eab0e74 rejected from java.util.concurrent.ThreadPoolExecutor@7d24538f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2773] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,578 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33315,1679676819242] 2023-03-24 16:56:40,579 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33315,1679676819242; numProcessing=1 2023-03-24 16:56:40,580 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33315,1679676819242 already deleted, retry=false 2023-03-24 16:56:40,580 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33315,1679676819242 expired; onlineServers=0 2023-03-24 16:56:40,580 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39701,1679676818001' ***** 2023-03-24 16:56:40,580 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-24 16:56:40,580 DEBUG [M:0;jenkins-hbase4:39701] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64cb50a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-24 16:56:40,580 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39701,1679676818001 2023-03-24 16:56:40,580 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39701,1679676818001; all regions closed. 2023-03-24 16:56:40,580 DEBUG [M:0;jenkins-hbase4:39701] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 16:56:40,581 DEBUG [M:0;jenkins-hbase4:39701] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-24 16:56:40,581 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-24 16:56:40,581 DEBUG [M:0;jenkins-hbase4:39701] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-24 16:56:40,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679676820773] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679676820773,5,FailOnTimeoutGroup] 2023-03-24 16:56:40,581 INFO [M:0;jenkins-hbase4:39701] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-24 16:56:40,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679676820775] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679676820775,5,FailOnTimeoutGroup] 2023-03-24 16:56:40,581 INFO [M:0;jenkins-hbase4:39701] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-24 16:56:40,581 INFO [M:0;jenkins-hbase4:39701] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-24 16:56:40,582 DEBUG [M:0;jenkins-hbase4:39701] master.HMaster(1512): Stopping service threads 2023-03-24 16:56:40,582 INFO [M:0;jenkins-hbase4:39701] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-24 16:56:40,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-24 16:56:40,582 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 16:56:40,582 ERROR [M:0;jenkins-hbase4:39701] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-24 16:56:40,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 16:56:40,582 INFO [M:0;jenkins-hbase4:39701] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-24 16:56:40,583 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-24 16:56:40,583 DEBUG [M:0;jenkins-hbase4:39701] zookeeper.ZKUtil(398): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-24 16:56:40,583 WARN [M:0;jenkins-hbase4:39701] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-24 16:56:40,583 INFO [M:0;jenkins-hbase4:39701] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-24 16:56:40,583 INFO [M:0;jenkins-hbase4:39701] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-24 16:56:40,584 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 16:56:40,584 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:56:40,584 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:56:40,584 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 16:56:40,584 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:56:40,584 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.83 KB heapSize=393.55 KB 2023-03-24 16:56:40,602 INFO [M:0;jenkins-hbase4:39701] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.83 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b74d459d025f4d1fa1db0ee748a43e81 2023-03-24 16:56:40,606 INFO [M:0;jenkins-hbase4:39701] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b74d459d025f4d1fa1db0ee748a43e81 2023-03-24 16:56:40,606 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b74d459d025f4d1fa1db0ee748a43e81 as hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b74d459d025f4d1fa1db0ee748a43e81 2023-03-24 16:56:40,609 INFO [M:0;jenkins-hbase4:39701] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b74d459d025f4d1fa1db0ee748a43e81 2023-03-24 16:56:40,609 INFO [M:0;jenkins-hbase4:39701] regionserver.HStore(1080): Added hdfs://localhost:40397/user/jenkins/test-data/84c50c72-d0c9-aaf8-3b8c-f0811ab7a641/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b74d459d025f4d1fa1db0ee748a43e81, entries=65, sequenceid=578, filesize=24.7 K 2023-03-24 16:56:40,610 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegion(2948): Finished flush of dataSize ~344.83 KB/353102, heapSize ~393.53 KB/402976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 26ms, sequenceid=578, compaction requested=false 2023-03-24 16:56:40,612 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 16:56:40,612 DEBUG [M:0;jenkins-hbase4:39701] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 16:56:40,614 INFO [M:0;jenkins-hbase4:39701] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-24 16:56:40,614 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 16:56:40,615 INFO [M:0;jenkins-hbase4:39701] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39701 2023-03-24 16:56:40,621 DEBUG [M:0;jenkins-hbase4:39701] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39701,1679676818001 already deleted, retry=false 2023-03-24 16:56:40,680 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 16:56:40,680 INFO [RS:0;jenkins-hbase4:33315] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33315,1679676819242; zookeeper connection closed. 2023-03-24 16:56:40,680 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@56b3a923 rejected from java.util.concurrent.ThreadPoolExecutor@7d24538f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2773] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,681 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): regionserver:33315-0x101177228be0001, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 16:56:40,681 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1e404b70] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1e404b70 2023-03-24 16:56:40,681 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ea740db rejected from java.util.concurrent.ThreadPoolExecutor@7d24538f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2773] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,682 INFO [Listener at localhost/43391] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-24 16:56:40,780 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 16:56:40,780 INFO [M:0;jenkins-hbase4:39701] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39701,1679676818001; zookeeper connection closed. 2023-03-24 16:56:40,780 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ef641e6 rejected from java.util.concurrent.ThreadPoolExecutor@7cb87fea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,781 DEBUG [Listener at localhost/43391-EventThread] zookeeper.ZKWatcher(600): master:39701-0x101177228be0000, quorum=127.0.0.1:62195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 16:56:40,781 ERROR [Listener at localhost/43391-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4611ef9 rejected from java.util.concurrent.ThreadPoolExecutor@7cb87fea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4635] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 16:56:40,782 WARN [Listener at localhost/43391] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-24 16:56:40,786 INFO [Listener at localhost/43391] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 16:56:40,895 WARN [BP-695438423-172.31.14.131-1679676814709 heartbeating to localhost/127.0.0.1:40397] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-24 16:56:40,895 WARN [BP-695438423-172.31.14.131-1679676814709 heartbeating to localhost/127.0.0.1:40397] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-695438423-172.31.14.131-1679676814709 (Datanode Uuid 508011da-6d8b-49c9-a04a-1fbd55f43f65) service to localhost/127.0.0.1:40397 2023-03-24 16:56:40,898 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e/dfs/data/data1/current/BP-695438423-172.31.14.131-1679676814709] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 16:56:40,899 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/62a7e4cc-7d06-05b9-9e80-15194e9bdb72/cluster_4c2d21f5-0e16-9296-ffe2-fafb659a211e/dfs/data/data2/current/BP-695438423-172.31.14.131-1679676814709] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 16:56:40,933 INFO [Listener at localhost/43391] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 16:56:41,105 INFO [Listener at localhost/43391] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-24 16:56:41,281 INFO [Listener at localhost/43391] hbase.HBaseTestingUtility(1293): Minicluster is down